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SUMMARY 


Objective 

The  overall  objective  of  this  project  was  to  design,  develop. 
Implement,  and  evaluate  an  authoring  system  which  would  provide  a  basis 
for  cost  effective  production  ( authoring )  of  computer-assisted  Instruc¬ 
tion  (CAI)  materials  for  use  In  the  context  of  computer-managed  Air 
Force  technical  training.  The  specific  target  application  was  the 
Advanced  Instructional  System  (AIS)  located  at  Lowry  AFB  and  the  pro¬ 
cedures  and  software  developed  were  to  be  Integrated  into  this  system. 

The  project  work  was  conducted  through  two  parallel  efforts.  The 
first  of  these  two  efforts  addressed  the  development  of  computer  soft¬ 
ware  to  facilitate  authoring,  presentation,  and  evaluation  of  CAI 
materials.  The  second  effort,  described  here,  concerned  the  definition 
of  a  practical  procedural  model  for  CAI  lesson  materials  production, 
development  of  a  CAI  Authoring  Procedures  Handbook,  and  evaluation  of 
the  complete  authoring  system. 

Approach 

Design  activities  began  with  analysis  of  the  probable  functions 
of  CAI  within  the  AIS,  review  of  prior  approaches  to  supporting  CAI 
materials  development,  examination  of  previous  military  CAI  development 
experiences,  analysis  of  the  characteristics  (training,  prior  experience, 
and  work  environment)  of  the  Air  Training  Command  (ATC)  personnel  who 
would  be  developing  CAI  materials,  and  re-examination  of  the  available 
AIS  software.  A  major  conclusion  resulting  from  these  analyses  was  that 
there  are  a  number  of  factors  In  the  military  technical  training  environ¬ 
ment  which  are  Incompatible  with  the  typical  approach  to  CAI  production 
and  that  prior  attempts  to  utilize  CAI  In  this  environment  had  not  taken 
these  factors  into  sufficient  consideration.  It  was  decided,  therefore, 
that  It  was  preferable  to  adapt  the  authoring  system  to  the  existing 
environment  rather  than  expect  the  environment  to  change  to  meet  the 
requirements  of  the  system,  even  when  this  approach  limited  the 
sophistication  of  the  CAI  materials  which  could  be  produced. 

The  most  comprehensive  component  of  the  authoring  system  Is  the 
procedural  model  for  CAI  development.  This  model  assumes  the  presence 
of  non-CAI  Instructional  materials  which,  for  one  reason  or  another, 
have  been  found  to  be  Inadequate.  It  begins  by  defining  criteria  for 
selecting  specific  lessons  for  CAI  development.  The  next  major  component 
Is  an  ISD-type  evaluation  of  the  selected  lessons  and  their  evaluation 
Instruments  In  which  any  deficiencies  In  content  or  tests  are  corrected. 
Next,  a  general  Instructional  strategy  and  an  individualization  strategy, 
consistent  with  the  lesson  content  and  student  population,  are  defined. 
Materials  development  Itself  Is  largely  structured  by  the  Authoring 
Editor,  to  be  described  later.  Following  certain  required  actions,  such 
as  stating  objectives,  the  author  Inputs  text,  diagrams,  and  embedded 
and  objective  level  test  questions  and  defines  performance-contingent 
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branching  logic.  The  use  of  appropriate  response  feedback  and  prompts 
(hints)  Is  emphasized.  Following  review  by  other  subject  matter  experts, 
one-on-one  student  tryouts  are  conducted  and  appropriate  revisions  made. 
Materials  are  then  Implemented  In  the  classroom  for  formative  eval¬ 
uation.  Standard  data  analysis  reports  are  used  to  detect  specific 
errors  and  appropriate  revisions  are  made.  On  the  basis  of  formative 
evaluation  data  and  past  experience  with  the  main-line  materials,  a 
student  assignment  selection  rule  Is  defined  and  Implemented.  The 
effectiveness  of  this  rule  Is  contrasted  with  random  assignment  and 
revised  as  necessary.  Finally,  the  overall  effectiveness  of  the  CAI 
materials  and  assignment  rule  are  periodically  monitored  and  revisions 
made  as  necessary. 

The  details  of  the  software  portion  of  the  system  have  been  re¬ 
ported  elsewhere  and  are  only  summarized  here,  briefly,  the  software 
consists  of  the  following:  The  Authoring  Editor,  which  allows  authors 
to  define  CAI  content  and  strategy  without  knowledge  of  a  programming 
language;  a  set  of  general  purpose  CAI  presentation  programs  containing 
automatic  data  collection  routines;  a  family  of  print  routines  for  list¬ 
ing  CAI  lesson  content  and  branching  strategy;  and  a  set  of  data 
analysis  reports  tailored  to  the  requirements  of  formative  evaluation. 

The  AIS  CAI  Authoring  Procedures  Handbook  describes  each  step  In 
the  authoring  procedures  model  and  provides  detailed  instruction  on  the 
use  of  the  Editor.  The  manual  Is  maintained  on-line  and,  hence,  is 
dynamic  In  nature  in  that  changes  In  documentation  of  the  authoring  pro¬ 
cess  can  be  made  and  distributed  In  near  real-time  fashion.  Up-to-date 
versions  of  the  manual  can  be  printed  and  distributed  as  required. 

Evaluation  Procedures  and  Results 

The  complete  authoring  system  was  evaluated  through  (a)  contractor 
development  of  CAI  materials  for  six  lessons,  two  block-revlevy  lessons, 
and  two  block-remediation  lessons  In  the  AIS  Weapons  Mechanic  course, 

(b)  Implementation  and  evaluation  of  the  instructional  effectiveness  of 
these  materials,  and  (c)  training  a  small  number  of  ATC  personnel  in 
CAI  authoring. 

The  CAI  materials  development  process  tel  lowed  the  steps  outlined  by 
the  procedural  model  and  employed  the  Authoring  Editor.  Hone  of  the 
three  members  of  the  contractor's  authoring  team  had  prior  CAI  exper¬ 
ience;  although  all  were  experienced  technical  training  authors. 
Approximately  2200  work  hours  were  required  for  development  of  the  six 
first-pass  modules.  These  modules  accounted  for  a  total  of  approximately 
25  Plan  of  Instruction  (POI)  hours  and  resulted  in  an  average  student 
contact  time  of  13.7  hours.  Thus,  development  required  an  average  of 
33  man  hours  per  POI  hour  and  113  man  hours  per  student  contact  hour. 

This  compares  very  favorably  with  the  figures  of  222  and  246  work  hours 
per  contact  hour  reported  by  other  researchers  for  military  technical 
training  CAI. 


The  Instructional  effectiveness  of  the  CAI  modules  produced  was 
evaluated  by  contrasting  them  with  conventional  materials  which  they  re¬ 
placed.  The  six  first-pass  modules  (a)  reduced  average  lesson  failure 
rates  from  23%  to  6%,  (b)  reduced  average  lesson  objective  failure  rates 
from  19%  to  21,  (c)  reduced  average  first-attempt  study  time  by  4.9%, 

(d)  reduced  average  lesson  time  to  criterion  by  11.2%,  and  (e)  reduced 
the  average  end-of-block  test  objective  failure  rate  from  16%  to  10%. 

The  two  block  review  modules  reduced  the  average  end-of-block  test 
objective  failure  rate  from  15%  to  11%.  The  CAI  block  remediation 
modules  were  not  evaluated  due  to  the  small  number  of  block  test 
failures  during  the  evaluation  period. 

Three  ATC  Instructors  were  trained  In  use  of  the  authoring  system 
during  15  one-half  day  sessions.  None  of  the  trainees  were  computer 
programmers  or  had  any  prior  CAI  development  experience.  There  was  no 
formal  training  after  the  first  session.  Rather,  each  trainee  was  given 
a  copy  of  the  Authoring  Procedures  Handbook  and  Instructed  to  use  the 
authoring  system  to  develop  a  CAI  module  approximately  one  student  con¬ 
tact  hour  In  length.  Contractor  personnel  were  available  to  answer 
questions  and  review  the  trainee's  work.  At  the  end  of  the  training 
period,  each  had  developed  a  module  through  the  stage  of  revision 
following  single  student  tryouts.  The  author  trainees  asked  relatively 
few  questions  and  the  modules  produced  were  of  generally  good  quality 
and  capitalized  on  the  capabilities  of  CAI.  The  trainees  were  quite 
satisfied  with  the  authoring  system  and  all  expressed  Interest  In  Im¬ 
plementing  CAI  In  their  courses. 

Concl uslons 


The  approach  taken  to  facilitating  CAI  development  appears  very 
promising.  Experience  to  date  has  demonstrated  that  effective  CAI  can 
be  produced  by  non-proqranmers  at  a  very  reasonable  cost,  comparable  to 
the  effort  required  to  produce  paper-and-pencll  materials.  ATC  per¬ 
sonnel  Teamed  to  use  the  authoring  system  and  produced  CAI  materials 
within  a  very  short  period  of  time.  These  trainees  expressed  highly 
favorable  attitudes  about  the  approach  and  found  no  faults  with  the 
system. 

The  report  concludes  with  a  number  of  recommendations  concerning 
further  development  and  use  of  the  CAI  authoring  system.  Proposed  im¬ 
provements  to  the  system  Include  a  dialog  for  detailed  author  definition 
of  response  processing  rules  for  anticipated  responses  to  constructed 
response  questions,  a  capability  for  author-generation  of  graphic  dis¬ 
plays,  additions  to  the  Handbook  addressing  Individualization  strategies, 
and  development  of  a  CAI  module  for  author  training  based  on  the  Hand¬ 
book.  Given  the  current  capabilities  of  the  authoring  system,  more 
extensive  use  of  CAI  Is  encouraged  for  the  current  AIS  courses,  and  It 
Is  suggested  that  further  applications  be  sought  In  other  resident 
courses  at  Lowry  AFB  and  In  the  area  of  detached  training.  Finally,  It 
Is  suggested  that  author  training  be  made  a  part  of  the  formal  Instruc¬ 
tor  training  course. 
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I.  INTRODUCTION 


Computer-based  Instruction  (CBI)  has  the  potential  of  achieving 
significant  savings  In  Air  Force  technical  training  related  costs  by 
reducing  course  lengths,  administrative  workloads,  etc.  A  number  of 
steps  toward  realizing  this  potential  have  already  been  taken,  e.g.,  use 
of  the  PLATO  computer-assisted  Instruction  (CAI)  system  and  Air  Force 
development  of  the  Advanced  Instructional  System  (AIS)  at  Lowry  Air 
Force  Base. 

The  AIS,  which  provided  the  context  for  the  work  described  here, 
was  a  large  scale  C8I  system  designed  to  Improve  the  effectiveness  and 
efficiency  of  Air  Force  technical  training  and  to  provide  an  operational 
research  facility  for  assessing  Innovations  In  Instructional  technology. 
The  system  supported  four  technical  training  courses  representative  of 
the  range  of  cognitive  and  performance  skills  required  by  enlisted  Air 
Force  personnel.  The  AIS  utilized  an  Instructional  decision  model 
employing  state-of-the-art  computer  hardware,  software,  statistical 
methodologies,  and  Instructional  management  and  Individual Ized  assign¬ 
ments  to  alternative  Instructional  materials. 

The  AIS  was  designed  to  support  both  CAI  and  computer-managed  in¬ 
struction  (CMI ) .  CMI  can  be  defined  as  a  situation  In  which  the  majority 
of  the  student's  Instructional  activities  are  completed  off-line.  The 
computer's  role  Is  that  of  evaluator,  diagnostician,  prescrlber,  and 
manager  of  Instructional  events.  In  CAI,  by  contrast,  all  of  the  stu¬ 
dent's  Instructional  activities  are  conducted  on-line,  at  an  Interactive 
computer  terminal.  CMI  can  be  characterized  as  being  extensive,  managing 
Instruction  for  a  large  number  of  students  throughout  a  large  body  of 
course  content.  CAI,  on  the  other  hand,  Is  typically  Intensive,  con¬ 
centrating  on  detailed,  highly  Interactive  Instruction  for,  generally, 
a  limited  segment  of  course  content  and  a  relatively  small  number  of 
students.  Extensive  application  of  CAI  has,  to  date,  been  restricted  by 
the  limited  utility  of  short  segments  of  Individualized  Instruction 
embedded  In  a  group-paced  environment  and  by  its  high  costs.  In  terms 
of  both  terminal  costs  and  materials  production. 

The  work  reported  here  capitalized  on  the  concept  of  CAI  embedded 
In  the  context  of  CMI.  Such  an  Integrated  system  permits  more  ff1c1ent 
use  of  CAI  since  student  pacing  has  been  Individualized.  In  addition, 
student  performance  on  CAI  lessons  can  be  recorded  directly  by  the  CMI 
system  and  the  extensive  student  performance  records  maintained  as  part 
of  CMI  can  be  readily  accessed  to  provide  truly  1r Jlvldual Ized  CAI  when 
and  where  It  Is  most  needed. 

Project  Purpose 

One  of  the  major  obstacles  to  widespread  adoption  of  CAI  has  been 
Its  relatively  high  cost  as  compared  to  more  conventional  Instructional 
media.  Traditionally,  the  high  costs  of  CAI  have  been  attributable  to 
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both  the  cost  of  the  computer  hardware  delivery  system  (l.e.,  the 
required  computing  power  and  the  need  for  an  Interactive  terminal  for 
each  student)  and  the  time  and  skill  levels  required  for  CAI  materials 
production.  Recent  years  have  seen  dramatic  reductions  In  the  cost  of 
computer  hardware,  particularly  with  respect  to  Interactive  terminals 
with  limited  graphics  capabilities.  Thus,  production  costs  have  be¬ 
come  a  major  factor  In  inhibiting  the  further  use  of  CAI  In  military 
training. 

There  are  a  number  of  reasons  why  CAI  production  costs  have  re¬ 
mained  high.  With  few  exceptions,  CAI  authors  have  been  required  to 
define  their  materials  and  Instructional  strategies  via  a  computer  pro¬ 
gramming  language.  Typically,  the  work  has  required  extensive  and 
detailed  effort.  For  the  CAI  produced  to  be  effective,  there  has  also 
been  a  requirement  for  relatively  sophisticated  Instructional  design  and 
evaluation  skills  specific  to  Interactive,  Individualized  Instruction. 
Due  to  these  requirements,  substantial  training  and  experience  have 
typically  been  necessary  before  CAI  authors  could  become  fully  pro¬ 
ductive.  The  objective  of  this  project,  therefore,  was  to  design, 
develop  and  evaluate  ar,  approach  to  CAI  authoring  which  would  allow 
novice  authors  to  be  Immediately  productive  In  CAI  materials  design, 
development,  evaluation,  and  revision. 

The  software  development  portion  of  the  project  has  been  described 
In  detail  by  Montgomery  and  Judd  (1979)  and  Is  only  summarized  here. 

The  purpose  of  the  work  described  in  this  report  was  to  define  a 
practical  procedural  model  for  selecting  promising  applications  for  the 
CAI  medium,  developing  materials  for  these  applications,  and  evaluating 
their  effectiveness;  to  develop  a  CAI  Authoring  Procedures  Handbook;  and 
to  evaluate  the  effectiveness  of  the  complete  CAI  authoring  system  In  an 
operational  environment. 

Project  Context:  The  AIS  Environment 

Each  course  supported  by  the  AIS  is  divided  Into  "blocks"  of  in¬ 
struction  which  may  require  from  1  to  10  days  to  complete.  Each  block 
contains  a  number  of  Instructional  units  or  lessons,  and  most  require 
a  comprehensive  end-of-block  test.  Within  a  block,  lessons  are  arranged 
In  a  hierarchy  based  on  their  prerequisite  relationships.  A  typical 
hierarchy  resembles  a  set  of  parallel  chains  diverging  and  converging  on 
certain  pivotal  lessons.  A  student  may  alternately  work  on  lessons  in 
two  or  more  chains. 

The  basic  unit  of  instruction  Is  the  lesson.  Each  lesson  consists 
of  one  or  more  objectives,  two  or  more  parallel  forms  of  a  criterion 
referenced  test,  criteria  defining  mastery  on  the  test,  and  typically, 
a  self-test  by  which  students  can  evaluate  their  understanding  of  the 
lesson  before  taking  the  criterion  test. 

A  lesson's  Instruction  Is  provided  by  one  or  more  modules,  each  of 
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which  teaches  the  complete  lesson  content.  Where  two  or  more  modules 
are  present,  they  represent  alternative  Instructional  treatments. 
Depending  on  the  lesson  content  and  the  nature  of  the  treatment,  a  module 
may  be  a  programmed  text,  an  elaborated  technical  order,  an  audio-visual 
presentation,  or  an  Interactive  CAI  session. 

An  AIS  Student  Scenario.  A  student's  first  experience  with  AIS  Is 
to  complete  a  preassessment  battery  consisting  of  a  number  of  scales 
which  assess  cognitive  and  affective  factors  considered  to  be  predictive 
of  a  student's  performance  In  the  course.  The  student  then  requests  his 
or  her  first  assignment  by  submitting  a  Forward-Going  Assignment  request 
at  a  management  terminal  which  consists  of  an  optical  scanner  and  medium 
speed  printer.  At  this  point,  the  student  Is  enrolled  In  the  course, 
but  has  not  yet  entered  a  block  containing  actual  course  content.  The 
system  then  selects  the  block  In  which  the  student  Is  to  start  work. 

Since  the  student  has  not  yet  completed  any  course  work,  only  blocks 
which  have  no  prerequisites  are  considered.  If  there  Is  more  than  one 
such  block,  the  one  containing  the  fewest  students  relative  to  the 
desired  number  in  that  block  Is  selected.  The  student  is  then  assigned 
to  an  appropriate  learning  center  and  home  carrel  and  to  a  specific 
lesson,  module,  and  criterion  test. 

Lesson  assignment  decisions  are  made  by  a  program  called  the 
Adaptive  Model  which  consists  of  two  major  components— the  Adapter  and 
the  Resource  Allocator.  The  Adapter  attempts  to  select,  for  each  assign¬ 
able  lesson,  the  one  module  which  Is  most  appropriate  for  that  student. 
This  decision  can  be  based  on  a  variety  of  rules,  e.g.,  select  the 
module  which  the  student  Is  predicted  to  complete  In  the  shortest  time 
given  that  the  student  Is  also  predicted  to  pass  the  criterion  test. 

Each  alternative  module  Is  given  a  weight  Indicating  Its  relative  pre¬ 
ference.  The  Resource  Allocator  assigns  preference  weights  to  modules 
on  the  basis  of  minimizing  the  assignment's  Impact  on  the  availability 
of  Instructional  resources.  Final  lesson  and  module  selection  Is  based 
on  a  compromise  between  the  two  sets  of  preference  weights.  The  form  of 
the  criterion  test  Is  selected  at  random. 

Having  received  the  first  assignment  printout  (called  a  Student 
Status  Report)  at  the  management  terminal,  the  student  reports  to  the 
learning  center  Instructor,  obtains  the  Instructional  resources  required 
for  the  assigned  module,  and  begins  work,  normally  at  a  home  carrel. 

After  studying  the  lesson  materials,  the  student  completes  a 
multiple-choice  self-test  and  reviews  the  material  pertaining  to  any 
questions  answered  Incorrectly.  The  student  then  completes  the  lesson 
criterion  test  and  submits  the  form  to  a  management  terminal.  The 
resulting  Student  Status  Report  details  the  student's  performance  on  the 
criterion  test  (percentage  total  score.  Items  missed,  objectives  failed, 
and  pass/fall  decision)  and  the  next  assignment.  If  the  test  criterion 
was  not  met,  the  student  Is  reassigned  the  same  lesson  and.  If  available, 
a  different  module.  After  restudying  the  lesson,  the  student  Is  assigned 
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another  form  of  the  test.  Otherwise,  the  lesson,  module,  and  test 
selection  procedures  are  repeated  and  the  student  Is  assigned  a  new 
lesson. 

If  the  student  assignment  Is  a  CAI  module,  there  Is  only  a  slight 
variation  In  these  procedures.  That  Is,  the  function  of  the  self-test 
Is  assumed  by  questions  embedded  In  the  CAI  presentation,  and  the 
criterion  test  Is  administered  on-line.  Test  results  are  submitted 
automatically  to  the  CMI  system,  and  the  Student  Status  Report  Is  dis¬ 
played  on  the  terminal.  A  printed  copy  of  the  report  Is  also  available 
from  the  management  terminal. 

When  a  student  has  completed  all  content  lessons  In  the  block,  a 
Block  Review  lesson.  If  required.  Is  assigned.  Following  review,  the 
student  Is  randomly  assigned  to  one  of  the  alternate  forms  of  the  block 
test.  While  lesson  tests  can  be  viewed  as  diagnostic  tools,  end-of-block 
tests  serve  a  certification  function.  That  Is,  since  there  Is  no  end- 
of-course  test,  block  test  performance  serves  as  the  basis  for  certifying 
mastery  of  the  objectives  contained  In  the  block.  A  student  who  does 
not  meet  the  block  test  criterion  Is  reassigned  to  the  block  In  a  status 
whereby  assignments  are  made  by  the  Instructor  rather  than  by  the 
system.  If  the  block  decision  Is  "Go,"  the  block  selection  logic  Is 
repeated  and  the  student  Is  assigned  to  the  next  block  of  study.  The 
student's  continued  progress  through  the  course  Is  essentially  a  rep¬ 
etition  of  these  events. 


II.  AUTHORING  SYSTEM  DESIGN  CONSIDERATIONS 

Design  of  a  cost-effective  CAI  component  for  the  AIS  began  with  an 
analysis  of  the  AIS  environment:  the  appropriate  role  of  CAI  within  a 
CMI  system  supporting  military  technical  training;  the  characteristics 
of  the  personnel  who  would  be  developing  CAI  materials;  and  the  software 
tools  currently  available  within  the  AIS.  Lessons  learned  from  prior 
approaches  to  CAI  development  were  also  considered. 

The  Role  of  CAI  Within  AIS 


First  It  must  be  recognized  that  the  prototype  AIS  Is  primarily  a 
CMI  system.  This  Is  not  to  Imply,  however,  that  the  system  was  not 
designed  to  acconmodate  CAI.  Rather,  within  the  context  of  the  AIS,  CAI 
was  seen  as  one  of  several  possible  media  available  for  Instructional 
purposes.  Management  and  monitoring  of  the  students'  progress  through 
a  course,  assignment  to  specific  Instructional  treatments  and  evaluation 
of  Instructional  effectiveness  are  all  supported  by  the  CMI  component 
of  the  system. 

Given  the  nature  of  the  AIS  form  of  CMI,  defining  the  role  of  CAI 
within  Its  structure  was  relatively  straightforward.  Recall  that  an  AIS 
course  Is  divided  Into  blocks  of  Instruction  which  conclude  with  certi¬ 
fication  tests.  Blocks  are  divided  Into  lessons  and  each  lesson  Is 
supported  by  one  or  more  modules,  each  of  which  addresses  all  of  the 
lesson’s  objectives.  When  two  or  more  modules  are  available  for  a 
lesson,  they  are  treated  as  alternative  Instructional  treatments  for 
that  lesson.  The  Adaptive  Model  assures  that  a  student  Is  not  assigned 
a  lesson  until  all  prerequisites  have  been  completed.  Assignment  of  a 
specific  module  Is  also  a  CMI  function.  Thus,  CAI  was  seen  as  provid¬ 
ing  one  of  two  or  more  alternative  Instructional  treatments  for  teach¬ 
ing  a  lesson,  and  as  such,  CAI  materials  were  to  be  packaged  and  assigned 
as  modules.  Student  terminals  required  for  CAI  were  to  be  treated  as 
Instructional  resources  managed  and  assigned  by  the  Adaptive  Model. 

It  was  assumed  that  If  a  student  was  assigned  and  completed  a  CAI 
module.  It  would  be  desirable  that  the  lesson  test  also  be  administered 
on-line  rather  than  via  a  management  terminal  Interaction.  While  the 
AIS  did  support  an  on-line,  computer-assisted  testing  (CAT)  capability, 
it  was  designed  primarily  for  block  tests  and  was  not  totally  suited  for 
administration  of  lesson  tests.  Therefore,  a  lesson-level  testing 
capability  was  to  be  Incorporated  Into  the  CAI  component. 

The  next  major  question  concerned  the  types  of  applications  for 
which  these  traditionally  expensive  modules  would  be  most  effective. 

While  the  use  of  CAI  for  normal  first-pass  Instruction  was  one  obvious 
answer.  It  was  hypothesized  that  the  branching  capabilities  and  moment- 
to-moment  control  over  student  behavior  afforded  by  CAI  would  be  par¬ 
ticularly  useful  for  the  functions  of  review  and  remediation. 
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As  AIS  courses  are  currently  structured,  nearly  all  blocks  end  with 
a  review  lesson  which  is  assigned  Immediately  prior  to  a  block  test.  No 
specific  Instructional  modules  had  been  developed  to  support  these 
lessons.  However,  the  CMI  system  provides  a  "Block  Report"  which  lists 
objectives  which  the  student  failed  on  the  first  attempt  at  lessons.  It 
was  Intended  that  students  would  use  this  list  to  review  those  block 
objectives  on  which  they  had  encountered  problems.  Instructional 
activities  during  this  period  are  determined  primarily  by  the  student 
and  the  Instructor.  Student  performance  data  from  these  activities  In¬ 
dicate  that  this  time  was  often  not  used  effectively.  CAI  was  seen  as 
an  excellent  way  of  remedying  this  situation. 

A  procedure  was  envisioned  In  which,  when  a  student  was  assigned  to 
a  block  review,  the  Adaptive  Model  would  assess  the  student's  prior  per¬ 
formance  In  the  block  and.  If  performance  was  found  to  be  marginal, 
assign  a  CAI  module.  The  CAI  module  would.  In  turn,  determine  the 
specific  objectives  on  which  the  student  had  encountered  problems,  re¬ 
view  the  student  on  these  objectives,  administer  and  evaluate  objective- 
level  diagnostic  tests,  provide  further  remediation  as  necessary,  and 
Issue  a  Student  Status  Report  suggesting  further  review  or  assignment 
to  the  block  test  as  appropriate. 

Block  remediation  presented  a  similar  situation.  A  student  who 
falls  a  block  test  Is  placed  In  a  "block  remediation"  mode.  In  this 
mode,  the  system  accepts  tests  Input  by  the  student  or  Instructor,  but 
the  system  does  not  make  specific  assignments.  This  role  Is  delegated 
to  the  Instructor.  To  guide  the  Instructor  In  making  appropriate  assign¬ 
ments,  the  objectives  which  the  student  failed  on  the  block  test  are 
listed  on  the  Student  Status  Report  printed  when  the  block  test  was 
scored.  Again,  student  performance  data  suggested  that  this  remedial 
time  could  be  employed  more  effectively. 

CAI  block  remediation  modules  were  envisioned  which  would  differ 
only  slightly  from  the  review  module.  Student  assignments  to  CAI 
remediation  would  be  made  by  the  instructor  and  selection  of  specific 
objectives  for  remediation  would  be  based  on  the  student's  block  test 
performance  rather  than  on  performance  on  the  block. 

Although  block  review  and  remediation  were  seen  as  two  prime 
targets  for  CAI,  It  was  assumed  that  CAI  would  also  be  used  for  alter¬ 
native  modules  for  first  pass  Instruction.  While  CAI  might  occasionally 
be  used  for  the  first  module  developed  for  a  lesson.  It  was  expected 
that  It  would  more  often  be  used  as  an  alternative  treatment  designed 
to  remedy  specific  problems  detected  In  an  existing  module.  For  example, 
some  students  could  benefit  from  Interactive  elaboration  of  concepts  In¬ 
stead  of  simply  being  presented  expository  remarks  which  might  only 
confuse  them.  Or,  It  could  be  used  to  present  frequent  questions  to 
assess  levels  of  understanding  and  thereby  effect  more  Individualized 
Instruction  through  appropriate  branching. 
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Anticipated  Characteristics  of  CAI  Development  Personnel 


In  considering  the  characteristics  of  the  personnel  who  would  be 
developing  CAI  modules.  It  was  first  thought  that  a  team  approach,  such 
as  advocated  by  Bunderson  (1973),  would  be  appropriate.  Such  an 
approach  would  specify  differing  roles  for  (at  least)  subject  matter 
experts.  Instructional  technologists,  and  program  coders.  However, 
further  analysis  of  the  military  technical  training  environment  strongly 
suggested  that,  while  desirable,  the  team  approach  was  very  likely  to 
encounter  serious  problems  In  practice. 

During  the  5  years  that  the  AIS  had  been  operational,  there  had 
been  repeated  efforts  to  define  specific  roles  for  specialists  who  are 
so  Important  to  effective  operation  of  CMI  (e.g.,  materials  writers, 
evaluators,  and  data  base  managers).  To  date,  these  efforts  have  had 
limited  success.  It  was  concluded,  therefore,  that  the  CAI  authoring 
system  should  be  structured  so  as  to  allow  a  team  approach  to  CAI 
development,  but  should  not  be  dependent  on  it. 

For  the  Inmedlate  future,  at  least.  It  was  reasonable  to  expect 
that  the  personnel  who  would  be  developing,  evaluating  and  revising  CAI 
materials  would  be  classroom  instructors.  It  was  assumed  that  such 
authors  would  typically  be  expert  in  their  subject  matter  area,  would 
have  limited  training  or  experience  In  Instructional  systems  design, 
would  have  no  prior  exposure  to  CAI  and  would  have  no  computer  program¬ 
ming  experience.  Additional  pertinent  characteristics  Included  a 
relatively  high  turnover  rate  for  military  Instructors,  little  or  no 
opportunity  for  formal  training,  and  limited,  fragmented  periods  of 
availability  (e.g.,  60  to  90  minutes  following  a  normal  Instructional 
day). 


An  obvious  problem  raised  by  this  profile  of  the  typical  CAI  author 
Is  the  lack  of  computer  programming  expertise  and  the  strong  Indication 
that  attempting  to  train  relatively  transient  authors  to  program  would 
not  be  practical.  Therefore,  an  approach  was  sought  which  would 
eliminate,  or  at  least  substantially  reduce,  the  need  for  computer  pro¬ 
gramming  on  the  part  of  the  author. 

A  second  major  Implication  of  the  analysis  of  authoring-personnel 
characteristics  was  a  need  for  procedures  which  would  structure  the 
authoring  task.  At  the  same  time.  It  was  recognized  that  excessive 
structuring  could  be  perceived  as  being  undesirable,  even  offensive, 
and  could  consequently,  detract  from  the  authors’  motivation.  It  was 
reasoned  that  they  would  be  more  amenable  to  task  structuring  once  CAI 
was  established  and  authoring  problems  had  been  recognized.  It  was  con¬ 
cluded  that  the  authoring  system  should  Include  a  set  of  general  pro¬ 
cedures  which  at  least  defined  and  outlined  each  critical  step  In  the 
development  process. 

The  high  turnover  rate  of  military  personnel  had  the  obvious 
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Implication  that  the  authoring  system  and  procedures  should  be  such 
that  a  novice  author  could  quickly  be  brought  to  a  productive  level  of 
proficiency.  Further,  It  was  apparent  that  It  would  often  be  necessary 
for  new  authors  to  complete  CAI  modules  that  had  been  designed  and 
partially  developed  by  others. 

Coupled  with  the  high  turnover  rate  was  the  expectation  that  there 
would  be  little  or  no  opportunity  for  formal  training  In  either  the 
design  or  the  mechanics  of  authoring  CAI  materials.  It  was  assumed  that 
most  training  would  have  to  be  conducted  on  the  job.  Thus,  self- 
Instructlonal  methods,  with  some  minimal  assistance  from  experienced 
authors,  appeared  necessary. 

Finally,  experience  gained  from  various  DOD  projects  Indicates  that 
management  of  Instructional  materials  development  Is  an  area  that  has 
been  generally  problem  prone  because  of  Its  complex,  unstructured  nature. 
Few  effective  management  procedures  have  been  successfully  Implemented. 
Consequently,  little  monitoring  of  Individual  authors  during  materials 
development  has  been  possible,  especially  with  respect  to  productivity 
and  quality  control.  Thus,  there  was  a  need  for  procedures  and  soft¬ 
ware  tools  which  would  facilitate  management  of  the  development  process. 
As  was  the  case  for  structuring  the  authoring  task  Itself,  It  was 
necessary  that  these  management  tools  be  fairly  open-ended  and  allow 
for  further  development  as  management  procedures  evolved.  Review  of 
completed  materials  by  other  subject  matter  experts  was  known  to  be 
particularly  time  consuming  and  fraught  with  problems  of  conflicting 
Interests  and  personal  opinions.  A  procedural  model  and  software  tools 
which  would  structure  and  accelerate  the  review  stage  were  considered 
especially  Important. 

CAI  materials  evaluation  had  potential  for  problems  which  were  at 
least  equal  In  severity  to  those  of  authoring  per  se.  It  was  assumed 
that  the  CAI  authors  would  have  primary  responsibility  for  formative 
evaluation  of  their  own  materials  but  would  have  little.  If  any,  prior 
evaluation  experience.  As  a  result,  there  was  a  need  to  structure  stu¬ 
dent  performance  data  collection,  retrieval  and  reporting.  In  this 
case.  It  was  thouqht  that  there  would  be  little  negative  reaction  to 
over-structuring.  Thus,  It  was  desirable  that  these  steps  In  the  pro¬ 
cedural  model  be  quite  explicit  and  that  the  data  collection  process 
be  almost  totally  predetermined  to  result  In  standard  reports  tailored 
for  formative  evaluation. 

Software  Considerations 


Two  aspects  of  the  AIS  software  which  strongly  Influenced  authoring 
system  design  decisions  were  availability  of  the  CAMIL  programing 
language  and  past  experience  with  the  use  of  Interactive  data  base 
editors. 
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CAMIL  (Computer  Assisted/Managed  Instructional  Lanquage) 

(Pflasterer,  1978)  Is  a  higher  level,  qeneral  purpose  programming 
language  developed  as  an  integral  part  of  the  AIS.  The  language  Is 
oriented  toward  support  of  both  CAI  and  CMI  applications  and  was  designed 
to  be  used  by  personnel  with  a  wide  range  of  experience. 

While  the  AIS  CMI  functions  are  supported  by  CAMIL  software,  day-to- 
day  operation  of  the  CMI  system  is  controlled  through  a  set  of  data  base 
editors.  The  Intent  of  these  editors  Is  to  allow  course  personnel  with 
no  programming  skills  to  define  the  characteristics  of  their  courses  and 
establish  the  rules  by  which  student  assignments  are  made.  With  this 
approach,  software  changes  are  only  required  when  the  basic  operating 
philosophy  of  the  system  Is  altered.  Normal  operational  changes  in 
course  content  and  configuration,  resource  Inventories,  and  student 
assignment  selection  rules  are  made  by  changing  the  course  data  base  via 
the  Interactive  data  base  editors. 

Operational  experience  with  this  approach  has  been  quite  positive. 
Relatively  few  software  changes  have  been  necessary  to  meet  the  system's 
evolving  Instructional  requirements.  Despite  the  complexity  of  the  data 
base,  course  personnel  are  able  to  use  the  editors  to  institute  data 
base  changes  appropriate  to  their  needs.  This  Is  not  to  say  that  the  use 
of  the  editors  Is  totally  straightforward.  The  data  base  and  the 
Interactions  among  Its  components  are  complex.  While  all  of  the  editors 
prompt  the  user's  Inputs  to  some  degree,  extensive  prompting  was  some¬ 
times  sacrificed  In  favor  of  efficiency.  In  retrospect,  this  trade-off 
was  sometimes  Inappropriate. 

In  general,  the  Interactive  editor  approach  was  thought  to  hold  con¬ 
siderable  promise  for  facilitating  at  least  some  aspects  of  the  CAI 
development  process.  It  was  also  recognized  that  where  such  an  approach 
was  adopted.  It  would  be  desirable  to  provide  extensive  prompting. 

Prior  Approaches  to  the  Authoring  Problem 

A  growing  recognition  of  the  problems  associated  with  CAI  develop¬ 
ment  has  resulted  in  a  substantial  literature  addressing  these  problems 
and  proposing  alternative  solutions.  The  following  sections  provide  a 
brief  Introduction  to  the  concept  of  CAI  authoring  systems  and  a  summary 
of  two  Air  Force  experiences  In  CAI  development. 

Authoring  System  Considerations  and  Criteria.  Zinn  (l-)74)  lists 
four  criteria  for  assessing  the  effectiveness  and  utility  of  CAI  author¬ 
ing  languages:  reliability;  efficiency;  flexibility;  and  convenience. 
Many  of  these  same  criteria  can  also  be  applied  to  the  more  general  con¬ 
cept  of  an  authoring  system. 

Under  reliability,  Zinn  Includes  automatic  recovery  for  both  author 
and  student  following  computer  failure,  limiting  the  loss  of  authored 
material  and  limiting  the  domain  of  author  errors,  i.e.,  an  error  In  one 
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part  of  a  program  should  not  Impact  other  parts  of  the  program  or  other 
programs.  Efficiency  refers  to  both  the  time  required  for  authoring  and 
the  computer  time  required  to  translate  author  language  statements  Into 
executable  code.  Flexibility  considerations  Include  access  to  a  variety 
of  devices  and  alternative  modes  of  execution  or  conventions  and  the 
capability  of  adding  new  operators,  statements,  and  subroutines. 

Author  convenience  Is  treated  as  a  major  consideration.  Zlnn 
suggests  that  the  language  (or  the  programming  component  of  an  authoring 
system)  should  have  a  minimum  of  redundancy  and  Irrelevant  syntax,  e.g., 
the  program  listing  should  be  no  more  complex  than  the  author's  actual 
task.  There  should  be  provisions  for  alternative  authoring  styles,  e.g., 
while  many  authors  Indicate  a  preference  for  Interactive  entry,  others 
prefer  to  work  with  paper  forms  which  remind  the  author  of  system  capa¬ 
bilities  and  requirements.  With  respect  to  revising  an  existing  program, 
Zlnn  notes  the  advantages  of  on-line  editing  and  suggests  that  the  system 
should  provide  access  to  the  original  file,  use  straightforward  notation 
for  determining  changes  to  be  made,  and  confirm  that  changes  were 
accomplished.  In  testing  a  program,  the  author  should  be  able  to  begin 
execution  at  any  point  and  trace  through  the  program  using  labels  as  In¬ 
dicators  of  location.  The  language  notation  should  help  a  reviewer 
understand  the  Intent  of  the  Instructional  content  and  strategy.  Finally, 
access  to  system  capabilities  should  Increase  with  experience. 

Kaplow  (1975)  states  that  In  order  to  maximize  assistance  to  the 
author.  It  Is  not  sufficient  to  simply  add  authoring  aids  to  a  program¬ 
ming  language.  Rather,  the  total  software  component  of  the  CAI  system 
must  be  organized  around  this  goal.  He  then  describes  What  he  con¬ 
siders  to  be  the  basic  features  of  such  a  system. 

First,  the  system  should  provide  a  structured  format  to  help 
authors  organize  their  concepts.  The  current  working  unit  should 
always  be  Identified  and  the  author's  statements  should  refer  only  to 
this  unit.  The  system  design  should  make  It  explicit  that  a  CAI  pro¬ 
gram  Is  a  collection  of  Information  organized  so  as  to  be  amenable  to 
understanding.  The  computer  Itself  should  automatically  perform  many 
programing  functions  such  as  checking  structural  completeness  and 
finding  cross  reference  errors.  Finally,  the  system  should  not  require 
that  a  program  be  complete  before  It  can  be  tried  out. 

In  his  subsequent  discussion,  Kaplow  makes  a  number  of  additional 
critical  points.  The  system  should  be  tolerant  of  user  errors  and  point 
out  errors  at  the  time  they  are  made.  The  details  of  presentation 
should  be  defined  on  the  basis  of  the  Implication  of  the  author's  In¬ 
structions  rather  than  requiring  the  author  to  specify  actions  In 
detail.  The  system  should  help  the  author  keep  track  of  the  Inter¬ 
relationships  between  the  various  parts.  The  fact  that  It  Is  often 
easier  to  write  a  new  program  rather  than  to  modify  an  existing  one  Is 
particularly  unfortunate  considering  the  opportunity,  even  requirement, 
for  CAI  revisions  based  on  student  performance.  Since  the  source  of 
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this  problem  Is  usually  one  programmer's  difficulty  In  understanding  the 
structure  and  logic  used  by  another,  the  system  should  place  particular 
emphasis  on  the  ease  with  which  one  can  use  existing  material. 

Example  Authoring  Systems.  The  earliest  CAI  programs  were  written 
In  the  available  general  purpose  computer  languages.  Although  this  Is 
still  a  popular  approach  (e.g.,  the  extensive  use  of  BASIC),  there  was 
early  recognition  that  authoring  could  be  facilitated  by  languages 
tailored  to  the  particular  requirements  of  CAI.  Consequently,  there  was 
a  proliferation  of  CAI  authoring  languages--at  least  30  by  1973.  The 
last  decade  has  also  seen  experimentation  with  various  author  entry 
systems— approaches  which  are  relatively  Independent  of  a  specific 
language.  The  following  paragraphs  briefly  describe  an  example  of  a 
modern  CAI  authoring  system  (PLATO  IV)  and  some  approaches  to  developing 

The  PLATO  (Programmed  Logic  for  Training  Operations)  IV  system 
relies  heavily  on  the  power  of  the  TUTOR  language  (Sherwood,  1974).  The 
system  provides  for  Interactive  entry,  easy  trial  and  revision  of  code, 
and  Is  quite  responsive  to  authors'  needs  in  terms  of  display  time,  com¬ 
pilation  time,  and  diagnostics.  The  author's  task  Is  facilitated  by  a 
number  of  aids  such  as  on-line  access  to  reference  materials,  simple 
program  routines,  and  files  of  current  documentation  and  comments.  To 
a  large  extent,  the  PLATO  approach  to  authoring  Is  based  on  the  model 
of  an  author  who  Is  a  versatile  professor:  an  expert  In  the  subject 
matter;  an  experienced  teacher  with  sound  but  Innovative  Ideas  about  In¬ 
structional  presentation;  and  a  capable  programmer.  In  other  environ¬ 
ments,  where  the  authors  have  been  less  experienced,  less  skilled,  and/ 
or  less  motivated,  the  approach  has  not  been  proven  to  be  as  satis¬ 
factory. 

Uowsey  (1974)  describes  five  categories  of  approaches  to  building 
easy  author-entry  systems:  (a)  separation  of  logic  and  content;  (b) 
avoidance  of  any  authoring  language;  (c)  use  of  lesson  planning  and 
formatting  guides;  (d)  conversational  materials  generation;  and  (e) 
macro  systems.  The  approaches  In  each  category  tend  to  build  on  the 
concepts  of  the  prior  categories. 

Almost  all  of  these  approaches  employ  the  tactic  of  separating  In¬ 
structional  content  from  program  loqlc.  This  divides  the  authoring  task 
In  a  way  that  Is  particularly  amenable  to  a  team  approach.  Dowsey  notes 
chat,  'O  be  effective,  such  separation  requires  similarity  of  structure 
between  the  two  components. 

The  no-author  language  approach  not  only  separates  content  and 
logic  but  does  not  require  the  author  to  define  the  logic.  Only  the 
content  Is  specified.  In  the  form  of  frames  or  problem  categories.  This 
Is  then  acted  upon  by  a  lesson  generation  program  which  treats  the  con¬ 
tent  as  data  to  produce  Instructional  materials. 
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The  use  of  lesson  planning  and  formatting  guides,  while  requiring 
the  services  of  a  coder,  permits  the  author  to  cormunlcate  In  English. 
Typically,  the  author  defines  the  material  t.o  be  presented,  the  questions, 
the  expected  answers,  and  the  corresponding  courses  of  action  on  a 
standard  form  which  specifies  the  categories  of  Information  required. 

One  of  the  more  sophisticated  examples  of  this  approach  Is  Dowsey's  own 
COURSEMAKER,  designed  for  use  with  the  COURSEWRITER  III  language.  It 
Is  based  on  use  of  a  paper  form  which  Includes  a  presentation  section 
containing  the  material  to  be  displayed,  a  question  section  Indicating 
the  student  response(s)  expected,  a  decision  section  deflnlnq  whether  a 
branch  Is  to  be  taken  and.  If  so,  the  type  of  branch,  and  an  analysis 
section  containing  response  judging  rules. 

Conversational  materials  generation  represents  a  quite  different 
approach.  The  interactive  system  assists  the  author  by  eliciting  the 
content  and  logical  structure  of  the  lesson  through  a  natural  language 
conversation.  Palolan  (1974)  describes  one  example  of  this  approach. 

The  author  defines  the  program  structure  by  entering  a  sequence  of  action 
verbs.  The  system  checks  the  accuracy  of  the  sequence,  and  If  It  Is 
correct,  prompts  the  author  to  supply  the  text,  anticipated  responses, 
counter  names,  etc. 

Uowsey  sees  the  use  of  macro  routines  as  potentially  being  the  most 
powerful  approach.  In  using  such  a  system,  the  author  retrieves  a  pro¬ 
gram  sequence  of  code  and  specifies  arguments  which  complete  the  routine. 
One  characteristic  of  this  approach  Is  that  It  Imposes  a  definite 
structure  on  the  material  produced.  T ICC  IT  (atetton,  Volk,  &  Bunderson, 
1973)  Is  probably  the  best  known  example  of  a  macro  system.  TICCIT 
employs  a  single  Instructional  strategy,  oriented  toward  concept  learn¬ 
ing,  which  assunses  learner  control  over  sequence.  The  strategy  specifies 
how  the  subject  matter  should  be  structured  (e.g.,  rules,  examples, 
practice  Items)  and  even  suggests  the  appropriate  number  of  items  In 
each  category.  Thus,  most  of  the  author's  task  consists  of  molding  the 
contents  to  fit  the  strategy.  Planning  guides  are  used  to  format  the 
content  and  define  presentation  tactics  by  selecting  ai*>ng  available 
options.  A  macro  processor  then  converts  this  Information  (content, 
content  format,  and  strategy  option)  Into  computer  language. 

One  other  macro-oriented  approach  that  deserves  mention  Is  the  use 
of  Monoforms  (Schulz,  1975),  developed  for  use  by  military  authors  on 
PLATO  IV.  The  Monoform  macros,  written  In  TUTOR,  were  Intended  to 
facilitate  preparation  of  frequently  used  question  types  by  providing  a 
question  format  and  eliminating  the  need  for  the  author  to  understand 
TUTOR.  A  total  of  nine  macros  were  developed  for  multiple  choice,  con¬ 
structed  response,  and  matching  questions.  Within  each  question  type, 
the  Monoforms  differ  with  respect  to  variation  In  format,  type  of  feed¬ 
back,  and  for  multiple  choice  questions,  order  of  alternative  presen¬ 
tation.  There  are  also  a  number  of  options  within  each  Monoform.  The 
author  copies  the  desired  Monoform  and  follows  the  Instructions  (supplied 
In  the  form  of  program  conments)  to  supply  the  question  content  and 


tailor  the  TUTOR  commands  to  his  or  her  specific  requirements.  Schulz 
reports  that  the  use  of  Monoforms  reduced  the  2-  to  6-hour  question 
development  time  to  only  10  to  15  minutes. 

Air  Force  CAI  Authoring  Experience.  Hlnwlch  (1977)  reports  a  com¬ 
parison  of  TlCClt  and  (^LAtO  authoring  efficiency  conducted  at  Maxwell 
AFB.  The  results  of  the  comparison  were  Inconclusive  with  little 
difference  found  between  the  two  approaches.  In  the  production  of  32 
contact  hours  of  CAI,  the  PLATO  team  required  an  average  of  222  work 
hours  per  contact  hour  while  the  TICCIT  team  required  246.  Of  greater 
Interest  Is  Hlmwlch's  description  of  the  procedures  followed  and  the 
problems  encountered,  particularly  by  the  PLATO  team.  The  training  pro¬ 
vided  for  PLATO  team  members  consisted  of  an  Intensive  2-woek  session 
and  subsequent  continuous  consulting  support  by  the  PLATO  staff.  TICCIT 
team  training  began  with  7  weeks  of  familiarization  followed,  some 
months  later,  by  3  weeks  of  Intensive  training.  In  both  cases,  the  train 
Ing  required  appears  excessive.  Further,  Hlmwlch  reports  that.  In  some 
Instances,  authors  did  not  capitalize  on  PLATO'S  flexibility  with 
respect  to  Instructional  strategies  and,  consequently,  did  not 
demonstrate  the  system's  full  capabilities. 

Oallman,  UeLeo,  .'lain,  and  Glllman  (1977)  provide  a  comprehensive 
description  of  an  evaluation  of  the  use  of  PLATO  IV  for  Air  Force  tech¬ 
nical  training.  At  the  beginning  of  the  test,  conducted  at  Chanute  AFB, 
the  typical  PLATO  authoring  model  was  adopted  with  each  author  acting 
Independently.  Learning  TUTOR  was  found  to  occupy  a  major  portion  of 
the  author's  time.  Authors  had  varying  styles  and  quality  control 
standards,  and  as  a  result,  the  curriculum  was  fragmented  with  little 
continuity  between  lessons.  Dallman  et  al.  concluded  that  a  basic  flaw 
In  this  approach  was  the  unrealistic  assumption  that  the  materials' 
authors  were  experts  In  both  subject  matter  and  Instructional  practices. 

A  team  approach  was  subsequently  adopted  in  which  the  team  con¬ 
sisted  of  an  author,  subject  matter  expert.  Instructional  programmer, 
and  coder.  Although  this  was  a  distinct  Improvement  over  the  prior 
approach,  a  number  of  problems  were  still  encountered,  do  written 
procedures  were  defined,  only  Informal  understandings  among  team  members. 
This  resulted  In  time-consuming  coordination  problems  and  inefficient 
use  of  team  specialists.  Administrative  and  management  procedures  were 
never  well  defined  and  there  was  a  continuous  need,  never  resolved,  for 
better,  more  extensive  author  training. 

A  number  of  PLATO  features  were  found  to  be  Quite  useful.  On-line 
data  collection  routines  supplied  by  the  PLATO  staff  and  the  capability 
for  on-line  text  editing  were  both  considered  Important.  TUTOR  was 
adequate  for  the  site's  needs  with  authors  handling  the  simpler  aspects 
of  programming  and  coders  required  for  only  the  more  complex  portions. 
Only  a  few  of  the  more  experienced  authors,  however,  capitalized  on 
PLATO'S  instructional  flexibility.  Almost  all  of  the  lessons  produced 
employed  the  same  simple  tutorial  model.  The  report  authors  suggest 
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that  this  approach  was  followed  because  the  materials  were  easy  to 
prepare,  the  subject  matter  was  not  that  complex,  and  the  student  com¬ 
prehension  and  retention  requirements  were  low.  branching  was  used 
mainly  for  forced  review  and  TUTOR'S  Impressive  response  judging 
capability  was  seldom  utilized.  Only  about  21  percent  of  the  questions 
developed  employed  a  constructed  response  format.  Not  only  were  con¬ 
structed  response  questions  more  time  consuming  to  program,  students 
disliked  them  because  of  un familiarity  with  the  typewriter  keyboard  and 
because  of  the  requirement  to  recall  and  spell  newly  Introduced 
terminology. 

Hallman  et  al.  drew  a  number  of  conclusions  relevant  to  the  current 
project.  The  team  Drocess  was  found  more  efficient  and  effective  than 
Individual  authors.  Authors  did  not  exploit  PLATO's  full  capabilities 
due  to  resource  constraints,  lack  of  CAI  expertise,  and  Inadequate 
training  in  Instructional  programming  techniques.  Finally,  sooh1st1catr 
,.M  (  .  ■■'Itles  may  not  be  necessary  for  the  tyoe  of  tasks  and  level  of 
knowledqe  required  for  most  phases  of  military  technical  training. 

The  types  of  problems  encountered  at  Chanute  and  Maxwell  appear 
typical  for  military  technical  training.  Klmberlln  (1977),  describing 
the  status  of  project  AuACUS  at  Ft.  Gordon,  reports  5  to  6  month  silos 
In  the  full  implementation  of  CAI  courses.  The  major  problems  en¬ 
countered  were  reported  to  be  changes  in  the  Plans  of  Instruction  during 
CAI  development  and  the  fact  that  the  project  was  never  assigned  an 
adequate  number  of  Instructional  programmers. 

Design  Conclusions 

Since  some  aspects  of  the  military  training  environment  are  less 
than  Ideal  for  efficient  development  of  Instructional  materials,  par¬ 
ticularly  CAI,  It  was  concluded  that  it  was  best  to  adapt  the  authoring 
system  to  this  environment.  It  was  hoped  that  the  authoring  system 
itself  would  act  as  a  mechanism  which  would  encourage  desirable  changes 
within  this  training  environment  to  be  instituted. 

A  major  consideration  In  the  design  of  the  AIS  CAI  authoring  pro¬ 
cedural  model  was  the  fact  that  the  CAI  to  be  developed  was  to  serve 
as  a  component  of  a  larger  computer-based  training  system.  Thus,  It 
was  concluded  that  the  procedures  should  encompass  the  complete  pro¬ 
cess  by  which  CAI  was  to  be  Integrated  Into  the  system.  This  would  In¬ 
clude  selection  of  materials  for  CAI  development,  analysis  of  existing 
materials  and  evaluation  Instruments,  definition  and  evaluation  of  rules 
for  assigning  students  to,  CAI,  and  subsequent  monitoring  of  CAI 
module  performance,  as  well  as  the  more  conventional  steps  of  materials 
development,  evaluation,  and  revision. 

In  keeping  with  Kaplow's  (1975)  suggestion,  the  total  software  com¬ 
ponent  was  deslqned  with  the  major  goal  of  facilitating  CAI  materials 
development,  despite  the  fact  that  many  features  of  CAMIL  had  been 
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designed  expressly  to  support  CAI,  it  was  recognized  that  any  approach 
that  relied  on  an  author/programmer  had  little  chance  of  success.  On 
the  other  hand.  It  was  considered  unlikely  that  personnel  would  be 
assigned  to  such  a  specialized  function  as  CAI  computer  programming  In 
sufficient  numbers  to  adequately  supoort  many  authors.  This  assessment 
suggested  either  a  no-author-language  approach  or  a  macro  system 
approach.  The  danger  In  these  approaches  was  seen  to  be  that  they  could 
be  too  rigid  for  the  evolving  comnuter-based  technical  training 
environment.  Therefore,  the  design  of  the  software  component  of  the 
authoring  system  relied  heavily  on  characteristics  of  the  existing  CAMIL 
system  and  capitalized  on  prior  AIS  eA,/«r1ence  with  Interactive  data 
base  editors. 

The  software  component  was  seen  as  consisting  of  three  major 
sections.  The  first,  most  visible,  part  was  an  authoring  editor  by  means 
of  which  an  author  could  define  CAI  content  and  branching  logic.  The 
format  of  the  author's  Input  was  to  be  as  similar  as  possible  to  what 
students  would  actually  see.  The  editor  was  to  automatically  accomplish 
as  much  of  the  programming  detail  as  feasible  and  to  eliminate  the  need 
for  authors  to  even  be  aware  of  the  CAMIL  language.  The  second  part  of 
the  software  component  was  to  be  a  flexible  "template"  CAI  presentation 
program  which  would  support  a  class  of  CAI  modules.  Principles  which 
had  been  used  in  developing  the  AIS  Adaptive  Model  and  Its  data  base  were 
employed  to  make  this  program  as  flexible  as  possible  without  undue  com¬ 
plexity.  The  basic  concept  was  that  the  content  and  branching  logic 
defined  via  the  editor  were  to  be  treated  as  data  by  the  presentation 
program.  Finally,  to  support  evaluation  activities,  the  presentation 
program  was  to  cont.dii.  jtpCent  performance  data  collection  routines,  and 
a  set  of  standard  data  analysis  reports  tailored  to  formative  eval¬ 
uation  requirements  were  to  be  provided. 

Difficulty  in  assuring  adequate  author  training  was  accepted  as  a 
major  problem.  It  was  concluded  that  much,  if  not  all,  author  training 
would  have  to  be  Informal  and  take  place  on  the  job.  Implying  a  need  for 
sel f-lnstructlonal  materials.  It  had  been  intended  that  an  Authoring 
Procedures  Handbook  would  be  provided.  It  was  now  recognized  that  this 
Item  would  be  critical  and  should  address  the  full  development  process, 
starting  with  materials  selection,  as  well  as  materials  authoring  per  se. 
In  addition,  It  was  concluded  that  the  authoring  editor  itself  should 
structure  the  author's  task  and  be  as  self-instructional  as  possible. 

Since  management  of  the  CAI  development  process  was  also  seen  to 
be  a  major  problem.  It  was  concluded  that  the  authoring  system  should 
contain  tools  which  would  facilitate  tracking  and  monitoring  the  pro¬ 
cess.  Capabilities  which  would  expedite  subject  matter  expert  review 
were  considered  to  be  especially  Important.  Wherever  possible,  it  was 
thought  that  the  software  component  should  automatically  capture  and 
display  Information  concerning  development  activities. 
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Ill.  APPROACH 


The  overall  approach  taken  to  provide  a  CA1  component  within  the 
A1S  was  to  design  and  develop  a  CAI  authoring  system  comprised  of  three 
major  elements:  a  procedural  model  for  selection,  authoring,  and  eval¬ 
uating  CAI  materials;  computer  software  to  support  the  authoring  process 
and  CAI  presentation;  and  a  manual  detailing  the  use  of  the  model  and 
software.  The  mod®l  and  software  were  developed  In  parallel,  with 
aspects  of  each  Influencing  the  other. 

The  model  assumes  the  establishment  of  a  team  of  material  developers 
to  accomplish  course  development.,  recognizing  that  this  Is  seldom 
actually  the  case.  The  team  should  consist  of  material  authors,  subject 
matter  specialists  (SMSs),  and  educational  technologists  or  psy¬ 
chologists.  The  team  leader  can  be  named  from  this  group  or  can  be  one 
of  the  course's  training  or  curricula  officers.  Thp  team  should  be 
established  prior  to  choosing  course  blocks  for  which  materials  are  to 
be  developed.  The  mix  of  team  members  will  vary  across  courses  as 
dictated  by  course  needs,  the  team  leader,  and  material  complexity. 
Functional  responsibility  for  specific  steps  within  the  procedural  model 
should  be  assigned  by  the  team  leader. 

A  Procedural  Model  for  Developing  and  evaluating  CAI  Materials 

Definition  of  an  effective  and  efficient  CAI  authoring  procedural 
model  for  the  Air  Force  technical  training  environment  began  with  an 
analysis  of  the  characteristics  of  the  Instructional  environment.  This 
examined  specifically  the  environment  defined  by  the  courses  supported 
by  the  Alb,  and  the  types  of  Instructional  problems  for  which  CAI  could 
provide  solutions  In  this  environment. 

In  general,  the  procedural  model  and  support  software  were 
oriented  toward  production  of  a  broad  range  of  tutorial  and  drill  and 
practice  materials  for  both  Initial  (first-pass)  study  ami  subsequent 
review  and  remediation.  Once  a  general  approach  had  been  outlined,  the 
authoring  procedures  were  further  defined  and  refined  on  the  basis  of 
AIS  experience  with  CM,  and  knowledge  gained  from  review  of  other  CBI 
systems  and  CAI  authoring  systems. 

The  procedural  model,  as  derived,  consists  of  six  maior  components, 
each  with  a  number  of  constituent  steps.  The  components  are  selection 
of  target  content,  CAI  materials  development,  formative  evaluation, 
definition  and  evaluation  of  student  assignment,  rules,  summatlve  eval¬ 
uation,  and  on-going  evaluation.  The  full  model  Is  shown  In  Tlgure  1. 

The  model's  six  component  steps  are  described  in  the  following  paragraphs. 

belecjMim  of  Target  Content.  The  first  component  of  the  procedural 
model  (see  figure"*7)*  assumes  the  presence  of  a  main  track  of  non-CAl 
materials  and  defines  criteria  for  selecting  those  materials  for  which 
alternate  CAI  modules  are  to  be  developed.  The  steps  In  this  component 


Figure  2.  Selection  of  Target  Content. 
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ao  not  constitute  a  precise  algorithm  for  content  selection.  Rather, 
the  Intent  Is  to  Identify  a  number  of  factors  which  should  be  considered 
In  determining  target  content  selection;  questions  which  repeatedly 
examine  potential  benefits  In  the  light  of  the  anticipated  costs  of  these 
benefits.  Few  hard  and  fast  rules  can  be  defined,  and  value  judgements 
play  a  major  role  In  the  decision  process. 

The  first  step  In  the  content  selection  component  Is  to  select  a 
course  or  courses  where  the  use  of  CAI  will  be  beneficial  to  the  train¬ 
ing  process.  This  should  be  a  school-level  decision  based  on  consider¬ 
ations  such  as  student  throughput,  training  cost  per  student,  expected 
amounts  of  courseware  change,  levels  and  types  of  simulation  required, 
student  entry  skills,  courseware  graphics  requirements.  Instructor/ 
student  ratios,  and  types  of  training  materials  used. 

The  principal  consideration  must  be  to  choose  a  course  where  CAI 
materials  can  Improve  motivation,  and  where  the  unique  capabilities  of 
CAI  will  provide  needed  Individualization.  A  course  might  be  selected 
If,  for  example,  conventional  materials  are  resulting  In  relatively  poor 
field  performance  and  the  cost  of  CAI  Is  relatively  small  when  compared 
to  overall  training  costs.  Alternatively,  CAI  hardware  costs  may  be 
relatively  unimportant  If  CAI  is  determined  to  be  the  only  available 
method  for  replacing  an  Inadequate  module. 

After  selecting  a  course,  the  next  step  is  to  select  a  block  or 
blocks  of  Instruction  for  conversion  to  CAI.  In  this  step,  many  of  the 
factors  considered  In  course  selection  apply,  e.g.,  types  of  materials 
used,  expected  amounts  of  courseware  change,  costs  of  training  aids  and 
simulation  devices,  etc.  Beyond  these,  however,  two  primary  consider¬ 
ations  are  block  test  failure  rates  and  tinr  to  complete  the  blocks,  as 
well  as  the  variability  In  these  measures.  If  the  failure  rate  on  a 
block  test  is  high.  If  variability  of  scores  on  a  block  test  Is  high, 
or  the  time  to  complete  a  block  is  greater  than  expected,  then  the  In¬ 
dividualization  and  Improved  motivation  that  can  be  provided  through  CAI 
may  be  a  cost-effective  approach  to  Improving  training. 

Having  tentatively  selected  one  or  more  blocks,  specific  lessons 
must  be  selected.  Here,  the  selection  considerations  are  lesson  failure 
rates,  variability  of  lesson  mastery  test  scores,  subsequent  block  test 
failures  on  the  objectives  Included  In  the  lessons,  and  average  times 
and  variability  of  times  to  complete  the  lessons.  As  with  block  selec¬ 
tions,  such  factors  as  rate  of  change  of  materials  and  costs  and  types 
of  training  aids  and  simulation  devices  must  also  bp  considered. 

With  respect  to  variability.  If  lesson  mastery  test  scores  are 
highly  variable  with  many  students  scoring  well  above  t.hp  passing 
criterion  (which  Is  judged  to  be  adequate).  It  may  be  that  those  stu¬ 
dents  should  be  given  more  cursory  explanations  of  some  oortlons  of  the 
materials,  thereby  reducing  training  time.  High  variability  In  time  to 
complete  the  materials  may  point  to  a  subset  of  students  who  do  not 
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understand  portions  of  the  materials.  In  this  case,  CM  might  be  used 
with  provisions  for  dlagnoslnq  specific  problems  and  individualizing  the 
instruction  to  decrease  training  tine. 

Men  the  three  selection  steps  (course,  block,  and  lesson)  are 
completed.  It  must  be  determined  if  CA i  is  feasible  for  the  tentative 
selections.  Possible  reasons  for  determining  at  this  stage  that  CAI  Is 
not  feasible  include  low  student  flow,  necessity  for  graphics  or  sim¬ 
ulations  which  are  beyond  the  capabilities  of  the  available  authoring 
personnel,  or  too  f°w  lesson  selections  to  justify  hardware  or  develop¬ 
ment  costs. 

If  CAI  is  feasible  for  the  selected  lessons,  the  next  question  is 
whether  CAI  material  should  be  developed  for  first-nass  attempts  over 
all  thp  objectives  within  the  lesson  or  should  cover  only  problem  objec¬ 
tives.  If  the  materials  are  to  cover  only  problem  objectives,  then 
should  they  be  used  for  review  before  a  block  test,  or  to  prepare  for 
a  second  attempt  on  a  block  test  covering  only  the  objectives  failed? 

If  materials  are  to  be  written  for  problem  objectives,  solely  or  In 
combination  with  first-pass  lessons,  the  available  data  on  objective 
failure  rates  on  first-pass  lessons  and  on  first-attempt  block  tests 
must  be  examined. 

With  the  final  selection  of  lesson  and  objective  materials  made, 
the  author  Is  ready  to  begin  the  next  component  of  the  procedural  model, 
CAI  materials  development. 

CAI  Materials  Development.  Development  of  CAI  materials  should 
begin  with  a  careful  analysis  of  Instructional  requirements  as  described 
In  Steps  2  and  3  of  the  Air  Force  Instructional  System  Development  (1SD) 
process  (AFM  50-2,  1979).  This  will  ensure  that  the  finished  materials 
avoid  such  problems  as  omitting  necessary  content,  covering  objectives 
Incompletely  or  at  an  Incorrect  level.  Incorrect  or  Inadequate  emphasis 
on  particular  subjects  or  concepts,  incomplete  material  supplements, 
poorly  designed  teaching  strategies,  and  tests  which  do  not  address  the 
objectives. 

The  steps  in  the  CAI  materials  development  component  are  shown  in 
Figures  ia  and  3b.  Steps  1  and  2  Involve  evaluating  the  measurement 
tools  and  the  content  outlines,  respectively,  of  the  materials  selected 
for  CAI  development.  The  content  outline  Is  essentially  a  listing  of 
every  teaching  step  of,  and  general  Information  concerning,  the  material 
to  be  taught.  This  evaluation  is  conducted  In  order  to  determine  that 
the  outlines  are  complete  and  that  the  tests  adequately  address  mastery 
of  the  materials.  The  course  Specialty  Training  Standard  (STS)  specifies 
the  required  levels  of  student  understanding  and  Is  the  standard  against 
which  outlines  and  tests  are  evaluated. 

In  the  first  step,  evaluating  tests,  It  Is  Important  that  first-pass 
(lesson)  tests  and  retention  (block)  tests  have  face  validity  for  the 
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objectives  which  they  test  and  address  the  same  subject  matter  at  the 
same  difficulty  levels.  It  Is  also  Important  that  alternate  versions  of 
lesson  and  block  tests  cover  the  required  subject  matter  at  the 
appropriate  difficulty  levels.  That  Is,  lesson  tests  should  first  be 
evaluated  and  alternate  versions  should  be  cross-checked  for  equal 
difficulties.  Block  tests  will  then  be  evaluated  and  alternate  versions 
cross-checked  for  consistent  levels  of  difficulty.  Last,  lesson  tests 
will  be  compared  with  the  block  tests  to  make  certain  that  the  block 
tests  cover  the  same  areas  to  the  same  level  of  difficulty  using  the 
same  criteria  for  passing.  The  overall  aim  Is  to  Insure  the  level  of 
mastery  required  by  the  STS. 

In  Step  2,  evaluation  of  content  outlines,  the  primary  consider¬ 
ation  Is  whether  the  content  outline  Is  consistent  with  the  content  area 
established  by  the  learning  objectives  and  their  criterion  tests.  Since 
the  content  outline  Is  the  principal  element  which  will  guide  the  writing 
for  materials.  It  must  fully  represent  the  Intent  of  the  STS  to  assure 
adequate  field  performance.  Following  Steps  1  and  2,  any  revisions 
must  be  reviewed  by  subject  matter  specialists  to  Insure  the  technical 
accuracy  of  the  content  changes. 

Unless  these  first  two  steps  are  successfully  completed,  all  sub¬ 
sequent  efforts  may  be  jeopardized.  For  example,  if  the  measurement 
Instruments  are  such  that  they  Indicate  good  student  performance  but 
the  field  evaluations  Indicate  Inadequate  graduates,  CAI  development 
work  will  be  largely  wasted. 

Once  It  has  been  determined  that  the  measurement  Instruments  and 
lessor,  content  outlines  are  valid  and  complete,  materials  development 
can  begin  with  the  definition  of  the  Instructional  techniques  to  be 
employed.  This  Involves  determination  of  both  Instructional  and 
individualization  strategies.  Typically,  the  two  steps  would  be  con¬ 
ducted  In  parallel  since  the  approach  taken  In  either  one  could 
Influence  the  other. 

Definition  of  Instructional  strategy  Includes  determining  the  order 
In  which  various  objectives  and  components  of  objectives  should  be 
taught  and  the  level  of  mastery  to  be  demonstrated  on  each  component 
before  the  student  Is  allowed  to  proceed.  It  should  be  noted  that 
monitoring  the  student's  performance  on  a  step-by-step  basis  Is  much 
more  feasible  with  CAI  than  with  other  self-instructional  media. 

Although  no  one  approach  Is  specified,  the  model  does  advocate  that  the 
Instructional  strategy  employed  be  based  on  a  behavior  classification 
scheme  such  as  that  proposed  by  Gagne  (1970). 

Individualization  strategy  concerns  Involve  the  methods  to  be  used 
In  matching  the  moment-to-moment  Instructional  events  to  the  student's 
current  level  of  knowledge.  For  example.  Is  pretesting  appropriate  or 
should  emphasis  be  placed  on  posttesting  and  review  of  troublesome  con¬ 
cepts?  What  Is  the  appropriate  frequency  of  embedded  questions?  Should 
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prerequisite  concepts  be  tested?  Does  the  subject  matter  and  the  anti¬ 
cipated  level  of  students'  prior  knowledge  lend  Itself  to  the  use  of 
learner  control?  An  Individualization  strategy  should  be  defined  for 
each  objective,  considering  the  routes  which  students  should  take 
through  the  materials,  based  on  responses  to  carefully  selected  and 
located  questions.  With  a  carefully  designed  pattern  of  branching  for 
a  lesson.  It  Is  possible  that  no  two  students  will  see  exactly  the  same 
combination  of  materials,  questions,  feedback,  and  prompts. 

After  instructional  and  Individualization  strategies  have  been 
defined,  SMSs  should  review  the  stmt--  ;i  .  In  this  way,  the  personnel 
developing  the  strategies  {e.g.,  educational  technologists)  can  be 
certain  that  there  Is  no  inherent  illogic  in  their  teaching  plans  which 
might,  for  example,  clash  with  classroom  procedures  or  field  duties. 

After  the  foregoing  analysis  and  design  steps  have  been  completed, 
the  actual  materials  authoring  can  begin.  If  the  preceding  steps  have 
been  performed  well  and  properly  documented,  the  author  need  only  be  an 
SMS  with  some  teaching  exper1ence--that  is,  particular  Instructional 
design  skills  are  required  for  the  subsequent  steps. 

The  CAI  authoring  Itself  Is  done  via  the  CAI  Authoring  Editor 
(developed  durlnq  this  project  and  described  In  the  next  section.  Soft¬ 
ware  dements  of  the  CAI  Authoring  System),  iirlefly,  the  Editor  struc¬ 
tures  the  authoring  process  and  leads  the  author,  step  by  step,  through 
this  process  via  an  English  language  dialogue.  Even  If  an  author  Is  an 
SMS,  developed  materials  should  be  reviewed  by  peers.  When  the  review 
process  has  been  completed  and  the  author  and  Instructional  strategist 
are  satisfied  with  the  quality  of  the  product,  single-student  tryouts 
of  the  materials  begin. 

Within  the  procedural  model,  single  student  tryouts  are  subsumed 
under  the  development  phase  rather  than  being  part  of  formative  eval¬ 
uation.  This  is  because  the  purpose  of  the  tryouts  Is  not  so  much 
directed  at  evaluating  the  Instructional  effectiveness  of  the  materials 
as  It  is  at  locating  errors  of  content  and  logic  which  were  overlooked 
by  the  authoring  team.  Typically,  the  number  of  students  run  would  not 
be  sufficient  to  answer  questions  concerning  Instructional  effective¬ 
ness.  Such  single  student  tryouts  do  provide  an  avenue  by  which  the 
author  can  learn  If  the  materials  are  easily  understood  by  their  Intended 
audience.  They  provide  answers  to  questions  such  as  the  following.  Is 
wording  clear?  Is  order  of  progression  correct?  Are  directions  to  the 
student  clear  enough  to  be  followed?  Are  there  Inconsistencies  between 
concepts  and  their  examples?  Is  the  lesson  too  long  to  hold  the 
student's  attention?  Single-student  tryouts  should  generally  be  run 
with  several  (6  to  12)  students,  but  the  number  will  vary  with  the  number 
of  errors  found  early  In  the  tryouts  and  the  extent  of  the  branching 
In  the  module.  If  many  errors  are  found  with,  for  example,  the  first 
two  or  three  students,  it  would  be  preferable  to  suspend  further  tryouts 
until  the  problems  found  have  been  resolved.  If  the  individualization 
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strategy  Is  fairly  complex.  It  may  take  a  larger  number,  up  to  20  for 
example,  before  all  problem  areas  are  Identified. 


If  available  manpower  permits,  It  Is  desirable  to  supplement  the 
single-student  tryouts  with  review  by  other  Instructional  (or  clerical) 
personnel  for  the  purpose  of  locating  grammatical  and  typographical 
errors  and  problems  with  clarity. 

Selection  of  students  to  try  out  the  CAI  materials  should  be  on  a 
volunteer  basis.  Using  volunteers  rather  than  Individuals  ordered  to  the 
tryouts  will  avoid  some  possible  attltudlnal  problems.  Volunteers  should 
be  from  the  course  for  which  the  materials  were  developed  and  (usually, 
at  least)  should  have  completed  the  conventional  material  covering  the 
subject  matter. 

The  single-student  tryouts  should  be  conducted  outside  the  classroom 
and  should  be  Informal.  Attempting  to  work  within  the  classroom  Is 
likely  to  disrupt  regular  classroom  activities.  Authors  of  the  materials 
should  be  available  to  answer  questions,  provide  guidance  where  necessary, 
and  note  areas  where  students  have  difficulty.  In  addition  to  direct 
observation  by  the  author,  student  comments  should  be  collected  through 
the  system's  Comment  facility,  and  student  response  data  should  be 
collected  (these  features  are  described  In  the  section  entitled  Software 
Elements  of  the  CAI  Authoring  System).  The  comments  can  then  be  reviewed 
and  the  system-provided  response  analysis  report  can  be  used  to  verify 
suspected  problem  areas.  These  data  can  then  be  combined  with  comnents 
from  subject  matter  specialists  and  others  who  reviewed  the  material  to 
determine  required  revisions.  Such  minor  revisions  as  correcting  typo¬ 
graphical  errors  or  Incorrect  branches  can  be  made  as  soon  as  they  are 
Identified.  If  major  revisions  are  required,  the  revised  materials  must 
be  resubmitted  for  subject  matter  specialist  review  and.  In  some  cases, 
additional  single-student  tryouts  would  be  desirable  prior  to  formative 
evaluation. 

Formative  Evaluation.  Th*  formative  evaluation  component  Is  shown 
In  Figure  4.  As  opposed  to  the  single-student  tryouts,  the  purpose  of 
the  formative  evaluation  component  Is  to  evaluate  the  Instructional 
effectiveness  of  the  materials  and  to  determine  specific  areas  of 
weakness.  Furthermore,  the  formative  evaluation  activities  should  pro¬ 
vide  an  Indication  of  how  students  and  Instructors  will  accept  CAI  as  a 
medium  In  the  classroom  and  Indicate  the  presence  of  any  problems  In  the 
Interaction  of  the  materials  and  the  CBI  system  as  a  whole. 

The  first  step  In  this  component  is  Implementation  of  the  CAI  module 
for  Classroom  use.  This  Includes  a  number  of  requirements,  all  aimed  at 
making  the  CAI  materials  a  part  of  the  classroom  environment. 

The  classroom  must  be  physically  configured  to  accommodate  CAI 
terminals  and  to  handle  any  classroom  procedures  that  might  be  uniquely 
required  by  CAI.  Staff  and  student  briefings  should  be  held,  explaining 
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Figure  4.  Formative  Evaluation. 


33 


the  nature  and  purpose  of  the  to-be-lntroduce  1  CAI.  The  Instructors 
must  be  trained  in  CAI  operational  procedures  and  in  counseling  students 
on  CAI  lessons  and  use.  dackuo  materials  and  urocedures  must  be  Intro¬ 
duced,  to  be  ready  In  case  of  computer  failure.  One  or  more  members  of 
the  development  team  should  be  as s i  ,  t  !  *  <  t'i  ssroom  for  the  first 
several  days  to  observe  and  assist  in  the  transition  and  to  answer  stu¬ 
dents'  and  Instructors'  questions. 

The  CMI  system  data  base  records  must  be  changed  to  identify  tne 
CAI  terminals  to  the  Resource  Allocation  f1odel,  to  reflect  the  addition 
of  the  CAI  modules  to  the  lesson  records,  and  to  define  the  rules  by 
which  CAI  modules  are  to  be  assigned  to  individual  students.  Ideally, 
students  should  be  assigned  to  the  CAI  nodule  at  random  to  provide  a 
basis  for  subsequent  determination  of  assignment  rules.  Typically,  how¬ 
ever,  this  will  not  result  in  the  most  efficient  use  of  the  available 
terminals.  If  there  is  a  requirement  to  complete  formative  evaluation 
more  quickly,  an  assignment  rule  can  be  defined  such  that  a  student 
assigned  to  a  lesson  with  a  CAI  module  will  be  given  the  CAI  module  if 
a  terminal  is  available.  This  will  result  In  maximum  use  of  t.he  CAI 
module  and  will  speed  up  data  collection. 

The  next  step  pertains  to  data  collection.  Data  should  include  stu¬ 
dent  performance  on  the  CAI  materials  and  tests  (decision  point  data, 
response  point,  data,  and  listing  of  unantici nated  responses),  student  and 
instructor  comments,  classroom  observations,  and  (in  some  cases)  measures 
of  student  attitudes.  In  analvzing  the  data  judicious  constraints  may  be 
necessary  to  exclude  spurious  Information  arising  from  cases  of  mis- 
assignment,  lesson  overrides,  and  students'  failure  to  follow  procedures. 
Ttie  evaluation  sample  size  must  be  agreed  upon  (typically  30  to  50)  and 
target  dates  set  for  reaching  milestones  or  terminating  the  formative 
evaluation. 

The  formative  evaluation  Itself,  the  data  analysis  step,  should 
concentrate  on  the  standard  CAI  performance  analysis  reports  (described 
In  the  next  section)  suppl emented  by  classroom  observation,  instructor 
comments,  and  any  student  attitude  measures.  For  any  one  CAI  module. 

It  Is  most  efficient  to  begin  analysis  of  student  performance  with  the 
decision  Point  Report  which  provides  an  overview  of  student  performance 
within  each  objective,  i.e.,  the  mean  number  of  questions  presented  and 
answered  correctly,  mean  elapsed  time  to  complete  the  objective,  and  the 
percentage  of  students  categorized  as  having  or  not  having  mastered  the 
objective.  Using  these  data,  the  author/evaluator  can  quickly  locate 
problem  areas  within  the  module.  The  Response  Analysis  Report  (contain¬ 
ing  mean  response  latencies  for  all  frames  and  a  detailed  accounting  of 
student  responses  to  individual  questions)  can  then  bp  uspd  to  more 
speclflcal lv  determine  the  nature  of  the  problems  contributing  to  areas 
of  Instructional  weakness.  That  Is,  which  specific  questions  ar° 
being  answered  Incorrectlv  by  a  large  proportion  of  the  students  and 
which  distractors  on  these  questions  are  being  incorrpctl/  selpcted? 

Is  the  oroblen  in  the  wordino  of  the  question  Itself?  If  not,  what  are 


the  characteristics  of  the  preceding  Instruction  which  have  led  to  these 
errors?  Where  constructed  response  questions  are  employed.  It  will  be 
necessary  to  supplement  the  Response  Analysis  Report  with  use  of  the  Un¬ 
anticipated  Response  Listing. 

Hopefully,  major  revisions  to  the  materials  will  not  be  required 
following  formative  evaluation.  Single-student  tryouts  should  have 
uncovered  most  of  the  more  pronounced  errors  and  problem  areas.  Any 
major  changes  to  tests,  materials,  or  classroom  procedures  which  are 
required  should  be  reviewed  by  SMSs  before  the  changes  are  implemented. 

Definition  and  Validation  of  Student  Assignment  Rules.  This  com- 
ponent,  fourth  In  order  of  the  Procedural  Model,  Is  required  by  the 
assumption  that  It  Is  not  cost  effective  to  assign  all  students  to  the 
relatively  expensive  CAI  modules.  While  development  costs  are  constant, 
delivery  costs  (l.e.,  terminal  and  central  computer  time)  are  usually 
sufficiently  high  that  the  CAI  modules  should  only  be  assigned  to  those 
students  who  will  benefit  the  most  from  the  more  Interactive  medium.  In 
some  cases,  there  may  actually  be  a  subset  of  students  who  will  learn  more 
efficiently  from  the  original  (non-CAI)  materials.  The  steps  comprising 
this  component  are  Illustrated  in  Figure  5. 

To  carry  out  this  "alignment"  of  students  with  Instructional  media, 
the  educational  technologists  must  analyze  the  formative  evaluation  data 
for  the  purpose  of  formulating  assignment  rules.  These  rules  may  be 
either  regression  equations  or  heuristics  and  should  be  derived  from 
Information  as  to  what  type(s)  of  students  performed  best  on  CAI  as 
opposed  to  the  original  materials.  The  performance  criteria  should  In¬ 
clude  times  to  complete  modules,  test  scores,  and  (If  available) 
attltudlnal  scores.  The  rules  can  be  derived  directly  from  the  data,  or 
can  be  tempered  by  the  Introduction  of  factors  based  on  experienced  best 
guesses,  and  results  of  other  CAI  research.  When  the  rules  have  been 
defined,  authors  of  the  CAI  materials,  SMSs,  and  Instructors  should  be 
Informed  of  the  Intent  of  the  rules. 

The  assignment  rules  are  then  Implemented  In  the  data  base  and  Incor¬ 
porated  Into  classroom  procedures.  To  provide  a  control  group,  a  per¬ 
centage  of  students  (e.g.,  30  to  50  percent)  would  continue  to  be 
assigned  at  random.  Srleflngs  on  the  rationale  and  expectations  behind 
Implementation  should  be  given  to  Instructors,  staff,  and  students.  In¬ 
structors  In  the  classrooms  affected  must  be  trained  to  handle  mlsasslgn- 
ments  and  other  problems  and  to  make  assignment  overrides  and  manual 
assignments.  Someone  must  be  designated  to  coordinate  problems  between 
the  classroom  and  the  group  Implementing  the  assignment  rules,  and  to 
enforce  coordination  and  configuration  control  among  the  various  groups. 

The  data  collected  to  evaluate  the  assignment  rules  would  Include 
student  performance  within  the  CAI  module  Itself  and  on  the  lesson  and 
block  tests,  times  to  complete  the  materials  and  tests,  student  and 
Instructor  comments,  classroom  observations,  and  (If  available)  student 
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Figure  5.  Definition  and  Validation  of  Student  Assignment  Rules 


attitude  measures.  There  Is  a  need  to  specify  the  hypotheses  to  be 
tested,  determine  data  collection  constraints,  set  time  tables  for  data 
collection  and  the  number  of  samples  to  be  rjathered,  and  determine 
how  conrients  and  observational  data  are  to  be  used.  Since  the  primary 
concern  Is  whether  the  assignment  rules  are  better  than  random  assign¬ 
ment,  data  from  both  groups  must  be  carefully  compared.  If  random 
assignment  proves  to  be  as  good  as  or  better  than  the  assignment  rules, 
the  rules  definition  process  should  be  rerun.  If  the  assignment  rules 
are  shown  to  be  superior.  Implementation  should  continue,  with 
periodic  rechecks  on  effectiveness. 

Summatlve  evaluation.  Whereas  formative  evaluation  Is  concerned 
with  the  Instructional  effectiveness  of  Individual  CAI  modules  and  with 
the  Implementation  of  the  materials,  suimiatlve  evaluation  Is  concerned 
with  the  Impact  of  the  CAI  modules  and  their  assignment  rules  on  the 
block  In  which  they  are  taught  and  with  their  Interaction  with  materials 
In  other  blocks.  The  steps  for  this  component  of  the  Procedural  Model 
are  shown  In  Figure  6. 

Since  no  changes  to  classroom  operations  or  the  data  base  are  re¬ 
quired,  no  additional  Implementation  procedures  are  necessary.  Thus, 
the  first  step  pertains  to  data  collection.  The  principal  concern  Is 
with  student  performance  times  and  scores.  The  data  of  primary  Interest 
are  module  completion  times  (for  both  the  CAI  module  and  the  original 
materials),  lesson  and  block-test  scores,  and  times  and  scores  on 
lessons  and  blocks  for  which  the  lesson  containing  the  CAI  module  Is 
considered  to  be  a  prerequisite.  Data  of  these  types  are  provided  by 
the  CMI  system's  standard  performance  reports.  Wlthln-module  (CAI)  data 
collection  Is  also  continued  however,  to  provide  detailed  student  per¬ 
formance  data  If  problems  are  uncovered  by  the  more  gross  CMI  measures. 
Little  emphasis  Is  placed  on  comments  or  classroom  observations.  Any 
general  classroom  procedural  problems  should  have  been  eliminated  during 
formative  evaluation.  Sample  size  should  approximate  50  students  per 
module. 

In  the  second  step  of  this  component,  CMI  performance  reports  are 
analyzed  to  determine  overall  changes  In  block  and  course  performance  as 
a  result  of  Implementation  of  the  CAI  materials.  Student  performance  on 
the  CAI  modules  should  also  be  re-examined  for  final  measures  that  are 
free  of  the  effects  due  to  Initial  implementation  and  change,  and  to 
the  novelty  of  CAI. 

If  the  preceding  steps  of  the  model  have  been  carefully  followed. 

It  Is  unlikely  that  serious  problems  will  be  detected  during  summatlve 
evaluation.  If  problems  are  encountered,  however,  they  may  be  attribut¬ 
able  to  the  module  assignment  rules,  the  Instructional  effectiveness  of 
the  CAI  module  Itself,  or  a  combination  of  both.  If  revisions  to  the 
assignment  rules  are  necessary,  the  Instructors  should  be  Informed  of 
the  changes  made.  If  major  revisions  to  the  module  are  required,  the 
SMSs  must  review  the  changes  made.  Any  significant  revisions  of  the 
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assignment  rules  or  the  module  requires,  at  least  in  part,  a  repetition 
of  data  collection  for  summative  evaluation. 


dhen  the  authors,  SMSs ,  and  evaluation  personnel  are  satisfied 
that  the  materials  are  satisfactory,  the  CA I  data  collection  routines 
can  be  turned  off.  There  is  no  need  to  continue  to  collect  detailed 
performance  data  for  C A I  materials  which  are  known  to  be  performing 
well. 


The  last  step  in  this  component  is  the  documentation  of  the  CAI 
module's  performance.  Tills  documentation  provides  a  standard  for  sub¬ 
sequent  (on-goinu)  evaluation  of  the  materials,  tests,  and  assignment 
rules  as  the  population  or  course  characteristics  change  and  provides 
comparison  data  for  subsequent  classes  of  students. 

On-Going  Evaluation.  As  the  student  population  or  course  and  field 
needs  change,  it  mav  be  that  student  performance  will  deteriorate.  The 
purpose  of  this  last  component  of  the  Procedural  Model,  On-Going  eval¬ 
uation  (see  Figure  /),  is  to  monitor  student  performance  and  to  address 
problems  which  are  detected  bv  this  monitoring. 

The  first  step  in  this  component  is  the  periodic  monitoring  of 
student  performance  data  collected  and  reported  by  the  C'JI  system.  The 
freguencv  of  such  monitoring  is  determined  by  the  rate  of  student  flow 
through  the  course  and  factors  such  as  how  often  student  or  field 
dynamics  change  and  whether  field  performance  is  becoming  inadeuuate. 
Typically,  monitoring  should  b«  conducted  at  least  quarterly.  The 
course's  evaluation  personnel  are  responsible  for  determining  which  data 
are  to  lie  examined;  what  constitutes  d  problem  (e.g.,  is  an  increase  of 
J  percent  in  the  failure  race  for  a  nodule  a  problem?);  and  how  many 
students  must  be  involved  before  a  problem  is  considered  to  be  sig¬ 
nificant. 

If  a  situation  involving  a  CAI  module  is  deemed  to  be  a  problem 
and  is  not  readilv  identifiable  as  arising  from  an  aumini strati ve,  non¬ 
computer  function,  toe  CAI  data  collection  routines  should  be  re¬ 
activated.  If  one  or  more  specific  portions  of  the  module  arp 
hypotliesi zed  to  bn  th“  source  of  the  problem,  it  mav  be  advisable  to 
define  decision  points  bracketing  the  area. 

As  in  formative  evaluation,  it  would  be  good  practice  to  again 
gather  student/instructor  comments  and  to  collect  at  least  limited  class¬ 
room  observations.  Comments  and  observations  mav  h°lp  to  separate 
attitudlnal  and  administrative  problems  from  those  internal  to  the  CAI 
modul es . 


'lata  analysis  and  interpretation  will  eventually  determine  whether 
the  problem  resulted  from  student  assionment  rules,  from  the  CAI 
materials,  or  from  a  rum-computer  administrative  function.  If  a  problem 
is  found  to  be  administrative,  actions  beyond  the  scone  of  r,,n  orocedural 
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model  are  required.  Otherwise,  revision  of  the  assignment  rules  or  the 
CA1  materials  Is  Indicated. 

If  an  assignment  rule  problem  Is  Identified,  the  steps  outlined  in 
the  "Definition  and  Evaluation  of  Student  Assignment  Rules"  component 
should  be  repeated,  remembering  that  Instructors  must  be  Inform d  of 
the  Instructional  Intent  as  assignment  rules  are  revised. 

If  the  problem  Is  Identified  as  being  within  the  CAI  module,  the 
problem  area(s)  should  be  revised,  following  the  revision  steps  out¬ 
lined  for  the  "Formative  Evaluation"  component  and  working  with  the  SMSs. 

In  either  case,  revision  must  be  followed  by  a  recheck  of  the  CMI 
and  CAI  data  following  the  changes.  If  the  problem  persists,  further 
analysis  to  more  completely  Isolate  the  nature  of  the  problem  Is 
required.  The  process  essentially  repeats  the  steps  of  data  analysis, 
revision,  data  analysis,  etc.,  until  the  problem  Is  resolved.  Once  the 
problem  has  been  satisfactorily  corrected,  the  CAI  data  collection 
routines  can  again  be  turned  off  and  periodic  monitoring  resumed. 

Software  Elements  of  the  CAI  Authoring  System 

The  development  of  computer  software  to  support  efficient  CAI  pro¬ 
duction  In  the  technical  training  environment  centered  around  construc¬ 
tion  of  a  CAI  Authoring  Editor  and  a  template  CAI  presentation  program. 
Student  performance  data  acquisition  routines  were  built  Into  the  pre¬ 
sentation  program  and  reports  were  developed  which  focused  on  formative 
evaluation  requirements.  A  CAI  Materials  Print  Program  was  developed 
to  provide  hard  copy  listings  of  CAI  materials  for  author  and  student 
use.  All  but  one  of  the  supporting  programs  were  written  in  CAMIL  and 
were  designed  to  operate  on  the  current  AIS  interactive  terminals  (a 
modification  of  the  PLATO  terminal)  but  provisions  were  made  for  easy 
transition  to  a  less  expensive  terminal. 

The  AIS  CAI  Authoring  System  software  does  not  require  the  author 
to  also  be  a  programmer  and,  due  to  the  use  of  extensive  author  prompting, 
reduces  the  need  for  protracted  author  training.  It  consistently 
reminds  the  author  of  system  capabilities  which  means  that  full  exploita¬ 
tion  of  the  system  Is  more  probable.  It  reduces  program  and  be- 
havorlal  debugging  since  content  and  strategy  are  treated  as  data. 

Further,  It  provides  for  meaningful  student  performance  data  collection 
and  reporting  which  is  both  structured  and  flexible. 

The  detailed  mechanics  of  the  System's  use  have  been  reported  by 
Montgomery  and  Judd  (1979)  and  will  only  be  summarized  here.  Since  the 
time  of  the  development  work  reported  here,  the  software  supporting  the 
Authoring  System  has  continued  to  evolve  and  expand.  The  software 
described  In  this  section  was  In  effect  at  the  time  of  the  work  reported. 

Authoring  Editor.  The  heart  of  the  software  supporting  the  AIS  CAI 
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authoring  system  Is  an  Interactive  Authoring  Editor— the  vehicle  by  which 
the  CAI  author  Interacts  with  the  authoring  svstem  to  produce  CM 
modules.  Rather  than  nrogrannlng,  the  author  defines  module  content  and 
logic  throuqh  an  English  language  dialogue.  Author  Inputs  are  in 
exactly  the  form  which  will  be  seen  by  students.  As  much  as  is  possible, 
the  Editor  structures  the  author's  task  of  developing  module  materials. 

At  each  Qoint  in  the  authoring  nrocess,  the  options  available  are  pre¬ 
sented  In  the  form  of  a  menu.  Selection  of  a  particular  option 
typically  leads  to  another,  more  specific  menu  of  options  for  executing 
the  selected  action.  In  some  cases,  as  in  defining  the  overall  structure 
of  the  i.todule  and  in  providing  certain  required  items  (e.g.,  a  statement 
of  the  objective),  the  author  is  forced  through  a  sequence  of  pro¬ 
cedures. 

The  actions  available  to  the  author  at  each  point  are  listed  as 
options  at  the  bottom  of  the  current  display.  An  author  can  select 
options  to  add,  delete,  cony,  save,  relocate,  and  °dit  materials;  pro¬ 
duce  simple  graohic  drawings;  gather  student/ reviewer  comments;  suggest 
student  breaks;  reorder  or  rcloca^  .i.'  i-id  presentation  sequences; 
define  specified  actions  to  bn  taken  by  the  Presentation  Program  (e.g., 
branches  and  data  coll  notion);  omit  presentation  of  specified  frames; 
and  determine  presentation  order  of  response  alternatives. 

As  structured  bv  the  Editor,  a  CM  nodule  is  divided  into  objectives 
which,  ideally,  correspond  to  t.hn  objective  listed  bv  the  Mr  Forc° 

Plan  of  Instruction  (P3I)  for  the  lesson  taught  bv  the  module.  An 
Objective  0  (zero),  containing  lesson  overview  material,  is  required 
and  the  author  mav  define  up  to  log  additional  objectives.  Thn  author¬ 
ing  system  is  frame  oriented  and  each  objective  can  contain  up  to  lhO 
frames,  each  of  which  may  consist  of  up  to  four  pages.  Three  classes 
of  frame  types  are  supported:  textual  content  frames,  question  frames, 
and  special  purpose  frames. 

At  the  time  of  the  development  work  reported  her°,  six  textual  con¬ 
tent  frame  types  were  recognized,  each  of  which  had  a  particular 
function:  main  line  text  presentation;  elaboration;  title;  statement  of 
objective;  overview;  and  listing  of  any  supplementary  materials  require¬ 
ments.  As  was  previously  stated,  the  author  entered  the  frame  content 
in  EXACTLY  the  format  in  which  it  was  to  bn  seen  by  the  student. 

Two  question  types  were  supported:  multiple  choice  and  constructed 
response.  All  formatting  of  question  frames  was  done  automatically. 

When  defining  a  question,  the  author  was  prompted  to  supply  the  question 
stem,  the  alternatives  or  anticipated  resoons°s,  a  feedback  statement 
for  “ach  alternative  or  constructed  response  (including  the  category  of 
unanticipated  responses),  a  prompt  (or  "hint")  statement  to  be  displayed 
after  each  successive  incorrect  attempt,  and  tno  number  of  attempts 
allowed.  Definition  of  at  least  on«  correct  answer  was  required.  Fronnt 
and  feedback  statements  w»re  optional  and  could  be  supnresse.j.  In  the 
case  of  feedback  statements,  the  author  could  also  elect  +o  have  a 
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standard  feedback  statement  selected  at  random  from  a  pool  for  presen¬ 
tation. 


The  special  purpose  frame  category  included  two  frame  types  - 
documentation  and  branching  decision  frames.  Neither  of  these  frame 
types  was  ever  presented  to  students.  Documentation  frames  were 
required  during  the  initial  creation  of  each  objective  and  were  intended 
to  provide  a  history  of  the  nodule's  development,  evaluation,  and 
revision.  Branching  decision  frames  allowed  an  author  to  define  a 
branching  point  without  requiring  that  any  frame  material  be  presented. 

The  author  was  able  to  define  three  types  of  branching  logic: 
loqic  evaluated  prior  to  frame  presentation;  logic  evaluated  following 
frame  completion;  or,  for  a  question  frame,  logic  evaluated  as  the 
student  responded.  Pre-  and  After-Frame  branching  logic  could  be 
"unconditional"  or  "conditional."  Unconditional  branches  were  always 
executed  if  the  student  reached  the  branching  point  while  conditional 
branches  were  made  only  if  ♦■he  student's  actions  matched  conditions  set 
by  fhp  author.  Thus,  the  author  could  specify  that  a  branch  be  taken 
conditional  on  (a)  a  specified  number  cf  an  author-defined  s»t  of 
questions  being  answered  correctly  or  (b)  incorrectly,  or  (c)  on  an 
author-specified  set  of  frames  having  been  or  (d)  not  having  been  pre¬ 
sented.  Response  logic  was  onlv  unconditional  in  the  s°nse  that  the 
author-specified  branch  would  always  be  taken  If  the  student  selected 
the  specified  alternative  or  constructed  response.  Any  number  of  Pre¬ 
frame,  Response  Contingent  and  After-Frame  branching  Instructions  could 
he  entered  against  a  sinqle  frame.  Logic  statements  were  evaluated 
sequentially,  both  across  and  within  categories.  Thus,  within  each 
category,  logic  statements  were  evaluated  in  the  order  specified  by  the 
author  until  a  statement  was  encountered  for  which  the  specified  con¬ 
ditions  were  true.  That  branch  wa s  th°n  taken  even  though  the  conditions 
specified  bv  subsequent  statements  may  also  have  bpen  true. 

uranchinq  logic  was  entered  In  what  is  essentially  a  highly 
prompted,  multiple  choice  format.  The  resultant  instructions  were  then 
displayed  In  Lngllsh. 

Presentation  Program.  CAI  materials  devploped  via  the  Authoring 
editor  w «>re  delivered  to  students  bv  a  CAI  Presentation  Program.  This 
program  consisted  of  a  general  program  structure  and  a  set  of  support 
routines  driven  by  the  CM  module  description,  decision  logic,  and  text 
records  createJ  by  th°  Authoring  Lditor.  Through  the  use  of  this 
generalized  oroqran  structure  and  table-driven  approach,  a  wide  rangp  of 
computer-assisted  tutorial  and  drill  and  practice  Instruction  could  be 
presented  with  minimal  programming  effort. 

Three  variations  of  the  presentation  urogram  were  used  to  present 
(a)  lessons  assigned  on  the  student's  first  pass  through  a  block;  (b) 
block  review  modules  assigned  prior  to  a  student's  first  attempt  on  a 
block  test,  reviewing  material  for  those  objectives  which  the  student 
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failed  while  studying  the  block  materials;  and  (c)  block  remediation 
modules,  assigned  after  a  block  test  failure,  reviewing  those  objectives 
which  the  student  failed  on  the  test.  The  basic  skeletal  program  and 
support  routines  were  written  during  the  development  of  the  first-pass 
modules  and  required  the  most  extensive  design  and  development.  The 
programs  to  support  block  review  and  remediation  were  then  constructed 
by  slightly  modifying  the  main  loop  code. 

Presentation  to  a  student  began  by  presenting  the  material  In 
Objective  0  followed  by  the  Instruction  contained  In  the  first  numbered 
objective  In  the  series.  Frame  descriptions,  branching  logic  and  text 
records,  all  created  by  the  Authoring  Editor,  determined  the  sequence 
and  content  of  the  presentation.  For  both  constructed  response  and 
multiple  choice  format  questions,  students  were  required  to  continue 
answering  until  correct,  until  reaching  the  specified  maximum  number 
of  attempts,  or  until  a  particular  answer  resulted  In  a  Response  Contin¬ 
gent  branch. 

At  any  point  In  an  objective,  the  student  could  opt  to  review 
material  which  had  already  been  presented.  In  the  review  mode,  text 
was  displayed  In  the  normal  manner  and  questions  were  displayed  with 
the  student’s  answers  Indicated. 

After  a  period  of  Instruction  and  practice,  an  objective  typically 
ended  with  a  series  of  test  questions.  Given  the  criterion  that  a 
certain  number  of  the  questions  be  answered  correctly,  author-defined 
branching  normally  routed  the  student  to  the  end  of  the  objective  as  soon 
as  the  criterion  had  been  met.  If  the  student's  performance  was  below 
criterion,  the  student  did  not  normally  exit  the  objective  until 
troublesome  points  had  been  reviewed  and  retested  with  additional  test 
Items. 

Upon  exiting  each  objective,  the  Presentation  Program  encountered 
either  an  objective-passed  or  objective-failed  flag  set  by  the  author. 

The  student  continued  through  the  objectives.  In  sequence,  until  all 
objectives  had  been  presented  or  until  the  program  encountered  a  lesson- 
passed  or  lesson-failed  flag.  When  a  lesson  passed/failed  flag  was  en¬ 
countered,  the  program  generated  a  module  test  form  containing  a  list 
of  any  objectives  failed  and  a  lesson  passed  or  failed  designator.  The 
program  then  passed  this  form  to  the  main  AIS  CMI  management  program,  the 
Adaptive  Model.  The  Adaptive  Model  recorded  the  student's  performance 
on  the  lesson,  generated  the  student's  next  assignment,  and  displayed 
It  on  the  terminal.  The  student  was  then  logged  off. 

Over  the  course  of  a  long  module,  an  author  could  encourage  the 
student  to  take  one  or  more  breaks--to  exit  the  module  and  leave  the 
terminal  for  a  short  rest  period.  Module  Interruptions  could  also  occur 
as  the  result  of  computer  failure,  end-of-shlft,  or  breaks  for  meals. 

In  each  case,  the  student  could  log  off  or.  If  the  Presentation  Program 
did  not  receive  a  keypress  for  a  specified  period,  the  student  was 


44 


r 


assumed  to  have  left  the  terminal.  After  displaying  an  Inquiry  as  to 
whether  anyone  was  there,  the  student  was  logged  off  automatically.  If 
a  module  was  Interrupted  for  any  reason,  the  Presentation  Program  auto¬ 
matically  restarted  the  module  at  the  frame  on  which  the  Interruption 
occurred  when  the  student  logged  back  onto  the  terminal. 

Access  to  the  CAI  Presentation  Program  was  not  limited  to  students. 
Lesson  authors  and  reviewers  could  use  the  program  to  verify  the  module’s 
content  accuracy  and  to  view  It  from  the  student's  perspective.  Author 
and  reviewer  access  to  the  program  was,  however,  handled  by  standard 
AIS  program  access  methods  rather  than  being  under  the  control  of  the 
Adaptive  Model.  Having  accessed  a  particular  module,  the  author/ 
reviewer  could  override  the  frame  control  logic  and  request  presentation 
of  any  frame  within  an  objective  through  use  of  a  special  function  key. 

In  addition,  the  author/ reviewer  could  always  enter  comments  about  the 
material  being  presented.  Student  comments  were  elicited  and  accepted 
only  if  the  Student  Comment  Flag  had  been  set  to  Htrue.”  Completion  of 
a  module  In  author/ reviewer  mode  did  not  result  In  submission  of  a 
lesson  completed  or  failed  form  to  the  Adaptive  Model.  In  all  other 
respects,  the  user's  Interaction  with  the  Presentation  Program  was 
Identical  to  that  of  a  student. 

Data  Acquisition  and  Reporting.  To  provide  student  performance 
data  collection  "and  analysis,  a  CAI  Data  Acquisition  and  Analysis  system 
was  developed  which  consisted  of  four  major  components:  (a)  data  re¬ 
cording  routines  In  the  Presentation  Program,  (b)  a  Data  Collection 
Program  to  move  performance  data  from  disk  to  tape,  (c)  a  Data  Analysis 
Report  Program  which  generated  three  different  types  of  reports,  and  (d) 
a  Report  Submittal  Program  to  facilitate  users'  requests  for  specific 
reports. 

Whether  student  performance  data  and  comments  were  collected  during 
the  presentation  of  a  CAI  module  was  dependent  on  whether  the  author  set 
appropriate  data  collection  flags  for  that  module  via  the  Authoring 
Editor.  This  philosophy  of  limited  data  collection  was  adopted  to  avoid 
generation  and  storage  of  the  Ittmense  amounts  of  data  which  would  other¬ 
wise  occur.  The  Intent  was  that  data  be  collected  for  formative  and 
summatlve  evaluation  purposes,  but  not  during  normal  operations  except 
for  consciously  Initiated  sampling. 

Response  data  represented  the  most  detailed  data  category.  The 
data  were  collected  at  the  end  of  each  frame  presented  to  the  student, 
regardless  of  frame  type.  In  addition  to  Identification  of  the  student 
and  the  frame,  and  the  time  at  which  the  frame  was  encountered,  data 
collected  Included:  (a)  total  time,  in  seconds,  spent  on  the  frame, 
and  (b)  any  time.  In  seconds,  spent  In  review  mode  If  review  was  in¬ 
itiated  from  the  frame. 

If  the  frame  was  a  question  frame,  the  following  data  were  also 
collected:  (a)  number  of  attempts  made  to  answer  the  question,  (b) 
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number  of  the  alternative  selected,  by  attempt  number,  (c)  response 
latency,  by  attempt  number,  (d)  number  of  unanticipated  responses,  and 
(e)  the  text  of  up  to  five  unanticipated  responses. 

Decision  Point  data  were  collected  at  the  end  of  the  module,  at 
the  end  of  each  objective,  and  at  the  end  of  each  frame  against  which  a 
Decision  Point  Flag  had  been  set.  In  addition  to  assorted  bookkeeping 
Information,  the  following  data  were  collected: 

1.  Elapsed  time  since  the  last  decision  point. 

2.  Number  of  questions  presented  since  the  last  decision  point. 

3.  Number  of  questions  answered  correctly  since  the  last  decision 
point. 

4.  Number  of  branching  logic  decisions  processed  for  the  current 
frame. 

For  each  branching  logic  Instruction  processed,  the  data  Included  (a) 
branching  type  (Pre-frame,  Response  and  After-frame),  (b)  number  of  the 
Instruction  within  Its  type,  and  (c)  the  branch  actually  taken.  If  any. 

Four  different  CAI  Data  Analysis  Reports  were  available  to 
authors  and  evaluators:  the  Decision  Point  Data  Report;  the  Response 
Analysis  Report;  the  Unanticipated  Response  Report;  and  the  Comments 
Listing.  In  addition,  standard  AIS  CMI  reports  could  be  used  to  provide 
a  description  of  overall  module  performance.  All  reports  were  requested 
from  an  Interactive  terminal.  For  the  three  CAI  student  performance 
reports,  requests  were  submitted  via  the  CAI  Reports  Program  which 
prompted  the  user  for  the  report  request  parameters,  thus,  the  user  did 
not  need  to  learn  how  to  set  up  job  control  parameters. 

The  Decision  Point  Report  was  generated  from  data,  stored  either 
on  disk  or  tape.  In  the  Decision  Point  Data  File.  It  provided  a  summary 
of  student  performance  within  each  objective  and  within  those  Intra¬ 
objective  segnents  which  the  author  had  defined  by  setting  Decision 
Point  flags  at  the  beginning  and  end  of  each  segment.  Each  component  of 
the  report  contained  the  number  and  name  of  the  Decision  Point  frame; 
whether  the  data  reported  pertained  to  students'  first,  second,  or  sub¬ 
sequent  pass  through  that  point;  the  elapsed  time,  number  of  questions 
presented  and  number  answered  correctly  since  the  last  Decision  Point; 
the  branching  logic  evaluated  at  that  point;  and  the  number  and  percent¬ 
age  of  students  taking  each  branch. 

The  Response  Analysis  Report  Identified  the  number  and  name  of  each 
relevant  frame  and  whether  the  data  reported  pertained  to  students', 
first,  second,  etc.  pass  through  that  frame.  For  frames  other  than 
question  frames,  only  time  data  were  reported.  For  question  frames, 
a  matrix  format  was  used  to  present  student  performance  and  response 
latency  data  as  a  function  of  the  response  (multiple  choice  alternative 
or  constructed  response)  selected  on  successive  attempts.  The  margins 
of  the  matrix  provided  a  summary  of  student  performance  on  the  question 
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(total  oercentaqe  correct,  percentage  correct  by  attempt  and  total  time 
to  correct  response)  while  the  matrix  cells  provided  a  more  detailed 
picture  of  how  students  reacted  to  the  question. 

The  Unanticipated  Response  Report  listed  each  unique  unanticipated 
response  for  each  constructed  response  question.  Unanticipated  responses 
were  listed  In  order  of  frequency  of  occurrence  together  with  the 
number  of  times  which  that  particular  response  had  been  entered. 

Comment  Listings  were  requested  via  the  Print  Option  of  the  Author¬ 
ing  Lditor  (described  below)  rather  than  the  CAI  Reports  Program.  Frames 
against  which  comments  had  been  made  were  indicated  on  the  Editor's  Frame 
List  display.  The  user  could  request  that  comments  made  on  a  particular 
frame  be  displayed  at  the  terminal  or  that  comments  on  one  or  more 
frames  be  listed  on  the  central  line  printer.  The  comments  could  also 
be  purged  from  thp  file  as  they  were  displayed  or  listed. 

Print  Program.  As  authors  created,  reviewed,  and  revised  CAI 
modules,  it  was  often  useful  to  work  from  hard  copy  printouts  of  the 
module's  content  in  addition  to,  or  In  placp  of,  the  displays  provided 
by  the  Authoring  Lditor.  There  w®r°  also  instances  in  which  hard  copy 
printout  was  desirable  for  student  use.  A  fea  ur°  of  the  CM  Authoring 
SvstPM  was  the  capability  to  request  a  variety  of  printed  listings  of 
CAI  modules'  content  and  branching  logic.  The  Print  Program  queried  the 
author  for  the  desired  print  options  and  then  initiated  a  soecial  back¬ 
ground  (non-lnteractive,  low  priority)  program  to  produce  thp  print¬ 
outs.  Ther®  wore  four  ji fferent  types  of  printer  listinos  available  to 
authors,  ranging  from  summary  information  to  detailed  listings  of  frame 
contents.  Multiple  copips  could  be  obtained  of  each  tvoo  of  listing. 


At  the  most  general  level,  the  Module  Summary  Listing  provided  an 
ov»rvipw  of  all  of  the  CM  modules,  operational  or  under  development, 
currently  defined  ir  th°  lata  base.  The  infomation  provided  for  each 
nodule  included  the  nodule  identifier,  module  title,  author's  IT,  and 
the  number  of  objectives  defined  within  th<=  nodule. 

For  a  particular  module,  the  Frame  Summary  Listing  provided  an  over¬ 
view  of  content  of  individual  objectives.  The  information  in  this 
listing  was  essentially  the  same  as  the  Lditor'-;  Frame  List  Display. 

Lach  frame  In  the  objective  was  listed  by  number  and  frame  name.  The 
existence  of  anv  branching  logic  and  Frame  Fla  ]s  was  noted,  and  the 
maximum  number  of  attempt;  allowed  to  answer  questions  was  shown.  If  a 
frame  was  an  alias  (l.e.,  referenced  anotner  frame),  the  referenced 
frame  was  identified. 

The  most  frequently  used  printout  was  probably  the  Frame  Contents 
Listinn,  the  complete  printout,  by  *rame,  of  all  text  and  question 
material.  Such  a  listing  could  b°  request®  I  Air  an  entire  module,  an 
Individual  objective,  or  a  specified  ser.  of  frames.  The  materials  con¬ 
tained  in  a  textual  content  frame,  u>  to  the  full  four  pages  were 
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printed  on  a  single  printer  page.  For  a  Question  Frame,  the  printout 
Included  the  question  stem,  the  alternatives  or  anticipated  responses 
with  the  correct  answers  denoted,  and  author-supplied  feedback  and 
prompt  messages. 

Finally,  the  Branching  Logic  Listing  provided  a  hardcopy  listing 
of  all  of  the  branching  logic  defined  for  frames  within  an  objective. 

The  format  In  which  Information  was  presented  was  similar  to  the 
Editor's  Branching  Logic  displays. 

In  addition  to  the  various  author's  listings,  hard  copy  printout 
of  a  CAI  module's  content  could  be  requested  In  a  format  appropriate  for 
direct  use  by  students  as  a  programmed  text.  Special  Purpose,  Documen¬ 
tation,  and  Branching  Decision  frames  were  automatically  suppressed  and 
the  author  could  elect  to  suppress  any  other  specific  frames.  All  other 
frames  were  printed  In  the  order  In  which  they  occurred  In  the  module. 
Branching  logic  was  simply  Ignored.  There  are  a  variety  of  uses  for 
such  printouts.  They  can  be  used  as  hard  copy  backups  for  students 
assigned  CAI  modules  In  the  case  of  computer  failure.  They  are  useful 
to  Instructors  for  answering  the  questions  of  students  assigned  CAI 
modules.  Their  most  Important  function,  however,  may  well  be  as  a  first 
step  toward  the  on-line  development,  evaluation  and  revision  of  materials 
Intended  for  off-line  use. 

AIS  CAI  Authoring  Procedures  Handbook 

The  Procedural  Model  for  CAI  development  and  the  purpose  and  use  of 
the  CAI  Authoring  System  Support  software  were  documented  In  an  "AIS  CAI 
Authoring  Procedures  Handbook"  (Lewis,  1979).  The  purpose  of  this  Hand¬ 
book  was  to  provide  a  document  that  could  essentially  "stand  alone." 

That  Is,  It  was  Intended  to  be  an  Item  that  could  be  used  by  the  average 
Air  Training  Command  materials  developer  to  learn,  with  no  other  train¬ 
ing  aids,  to  produce  CAI  materials  suitable  for  ATC  classroom  use.  It 
was  to  Include  guidance  as  to  the  characteristics  that  make  a  CAI  module 
(or  any  instructional  material,  for  that  matter)  a  better  teaching  tool; 
describe  all  of  the  features  of  the  CAI  Authoring  System  and  how  they  are 
used;  and  provide  an  overview  of  what  CAI  Is  and  how  It  can  benefit  the 
student. 

The  Handbook  was  developed  using  the  CAI  Authoring  Editor  and  Is 
maintained  on-line.  Its  contents  are  available  in  a  number  of  formats 
through  the  Print  Program.  The  Handbook  covers  all  of  the  current 
functions  and  aspects  of  the  Authoring  System  and  Is  dynamic  In  nature 
in  that  revisions  to  the  Handbook  occur  In  near  real  tine  as  modifi¬ 
cations  are  made  to  the  Authoring  System.  This  attests  to  the  capability 
of  the  Authoring  System  to  provide  quick  turnaround  for  maintenance  of 
Instructional  materials. 

Initial  review  of  the  Handbook  was  conducted  by  contractor 
personnel,  AFHRL  representatives,  and  ATC  administrative  personnel  from 


4d 


the  three  AIS  courses.  Inputs  from  these  groups  were  evaluated  and 
necessary  revisions  made. 

The  Handbook  was  then  printed  and  given  to  ATC  personnel  represen¬ 
tative  of  the  expected  target  population.  They  were  asked  to  review  the 
material  for  clarity  and  to  make  any  suggestions  which  they  thought 
might  lead  to  a  more  understandable  product.  It  should  be  pointed  out 
that  during  Initial  review  and,  to  some  extent,  the  subsequent  target 
population  review,  the  Handbook  was  frequently  being  updated  to  Include 
changes  and  additions  to  the  Authoring  System.  Discussions  about  the 
Handbook  were  conducted  between  the  contractor  and  the  target  population 
reviewers  on  a  somewhat  regular  basis;  usually  once  or  more  a  week. 
Comments  from  the  reviewers  were  generally  favorable,  and  requests  for 
changes  were  minor  In  nature.  For  the  most  part,  their  suggestions 
were  Incorporated. 
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IV.  AUTHORING  SYSTEM  EVALUATION  PROCEDURES 


Evaluation  of  the  Procedural  Model  and  the  Authoring  System  soft¬ 
ware  consisted  of  two  major  components:  development  and  imnl ementation 
of  CAI  materials  In  one  of  the  AIS  courses;  and  training  of  ATC  Instruc¬ 
tors  in  use  of  the  Authoring  System.  The  procedures  followed  In  both 
components  of  the  evaluation  are  reported  here.  Times  required  for 
materials  development,  results  of  summatlve  evaluation  of  the  materials 
developed,  and  data  pertaining  to  author  training  are  reported  In  Section 
V  of  this  report. 

• 

CAI  Materials  Development 

As  outlined  by  the  Procedural  Model ,  the  CAI  development  process 
began  with  selection  of  target  content.  Three  types  of  modules  were 
defined  as  a  result  of  this  component  of  the  process--modules  Intended 
for  first-pass  Instruction,  for  block  review,  and  for  block  remediation. 
The  various  steps  in  the  materials  development  component  were  followed 
quite  closely  with  the  exception  that  contractual  time  constraints  did 
not  permit  single-student  tryouts  of  the  block  review  and  remediation 
modules.  Similarly,  the  first-pass  materials  were  subjected  to 
formative  evaluation  and  revised  on  the  basis  of  the  evaluation  results 
but  no  formative  evaluation  was  conducted  for  the  review  or  remediation 
modules.  Due  to  time  constraints,  no  attempt  was  made  to  define  and 
evaluate  student  assignment  rules.  A  partial  suinmatlve  evaluation  was 
conducted  for  the  first  two  module  types.  A  dramatic  reduction  In  block- 
test  failures  which  Imlted  the  available  sample  size,  precluded  eval¬ 
uation  of  the  block  ipmedlation  modules.  No  attempt  was  made  to 
evaluate  the  final  step  of  the  Procedural  Model--0n-Go1ng  Evaluation. 

Target  Content  Selection.  It  will  be  recalled  that  the  first  step 
In  the  Procedural  Mode1!  concerns  selection  of  the  target  content  for 
which  CAI  materials  are  to  be  developed.  Of  the  four  available  AIS 
courses.  It  was  decided  that  the  CAI  authoring  effort  should  be  con¬ 
centrated  In  the  Weapons  Mechanic  (WM)  course  because  that  course 
represented  an  Instructional  setting  where  reasonably  difficult  materials 
are  taught,  there  was  a  sizeable  student  throughput,  student  ability 
levels  were  relatively  heterogeneous,  and  it  appeared  that  much  of  the 
course's  content  would  remain  relatively  stable  for  the  foreseeable 
future.  Although  the  course  Included  a  large  number  of  performance  tasks, 
students  were  also  subject  to  a  heavy  reading  load  and  Informal  observa¬ 
tion  suggested  that  student  boredom  was  often  a  problem.  An  additional 
consideration  concerned  the  recent  results  of  WM  field  evaluation  data 
which  indicated  that  course  graduates  were  not  performing  field  tasks  as 
well  as  was  to  be  expected. 

One  drawback  to  the  WM  course  as  the  CAI  testbed  was  that  in  many 
Instances,  various  visual  aids  were  required  to  supplement  tne  reading 
material  on  lessons  prerequisite  to  performance  tasks.  At  the  time  of 
this  project,  the  Authoring  System  supported  only  a  rudimentary  form  of 
computer-generated  graphics.  It  was  assumed,  however,  that  this  require- 
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Thus,  the  CAI  to  be  developed  would  be  evaluated  In  an  area  where 
readers  may  become  bored  with  conventional  texts,  where  textual  media 
needed  to  be  supplemented  by  visual  graphics  or  photographs,  where 
difficult  concepts  (which  are  often  preregul si tes  to  dangerous  per¬ 
formance  tasks)  were  taught  to  a  wide  variety  of  student  abilities,  and 
where  there  was  an  Indication  that  current  Instruction  was  Inadequate. 

To  the  extent  possible,  cAI  development  was  to  capitalize  on  the  ex¬ 
pertise  of  the  WM  subject  matter  experts  who  had  helped  to  develop  or 
had  at  least  reviewed,  the  original  materials. 

The  first  step  In  selecting  specific  WM  blocks  for  CAI  Imple¬ 
mentation  Involved  examining  end-of-block  test,  failure  rates  and  com¬ 
pletion  tines  for  each  block  in  the  course.  This  was  done  through  the 
use  of  standard  MS  CMI  reports  (the  Course  (.valuation  Summary  anil  Test 
Item  [.valuation  report)  and  special  purpose  data  extraction  using 
t hr*  data  extraction  Program.  One  of  the  problems  i  oted  at  tills  time  was 
that  the  cintent  validity  and  reliability  of  many  of  the  block  tests 
were  not  as  satisfactory  as  would  be  desired. 

Since  none  of  the  block  tests  had  a  first  attempt  failure  rate  In 
excess  of  lb  percent  at  the  time  of  the  analysis,  the  additional 
criterion  of  anticipated  content  stability  was  coupled  with  block  failure 
rate  as  a  basis  for  block  selection.  Iwo  blocks,  comprised  of  materials 
covering  guns  and  armaments  basic  to  a  number  of  aircraft,  were  not 
expected  to  be  replaced  In  the  near  future.  Since  these  two  blocks, 
block  S  and  f>,  bad  relatively  high  failure  rates  of  H  and  Id  percent, 
respectively,  they  were  selected  for  the  CAI  development  effort. 

With  the  choice  of  these  two  blocks,  lesson  selection  began.  Only 
cognitive  content  lessons,  not  performance  lessons,  were  considered. 

The  primary  lesson  selection  criterion  was  a  lesson  mastery  test  failure 
rate  at  or  above  lb  percent.  Data  for  the  failure  rate  analyses  were 
obtained  from  the  Course  1  valuation  Summary  which  summarizes  student 
performance  hv  nodule,  lesson,  and  block.  Flip  calendar  period  from 
which  data  were  drawn  was  1  opt  ember  h>//  to  ,’H  February  M/;i.  An 
additional  criterion  was  that  the  lessons  selected  be  typical  of  WM  train¬ 
ing  requirements ,  particular! v  as  they  related  to  field  performance. 

A  total  of  six  lessons  was  selected,  four  In  block  b  and  two  In 
blucl  o.  logether,  the  lessons  covered  the  use  of  technical  orders, 
i on, ilet Ion  of  forms,  and  general  Instruction  on  the  workings  of  missiles, 
■inns,  rockets,  and  fuses.  According  to  the  course’s  Plan  of  Instruction 
the  six  lessons  accounted  for  a  total  of  approximately  .’b  class¬ 
room  hours.  The  four  block  b  lessons  and  the  two  block  b  lessons 
accounted  for  14  percent  and  ,d  percent  of  their  respective  block's  vm  »’ 

,  ns.  There  were  on  I v  two  other  blocks  In  the  course  in  which 
lessons  with  failure  rates  at  or  above  lp  percent  accounted  tor  a  sub¬ 
stantial  part  of  the  block  (lo  percent  and  ,d  percent).  When  the 
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lessons  selected  are  viewed  as  a  percentage  of  the  blocks'  cognitive 
content,  as  opposed  to  cognitive  plus  performance  content,  the  per¬ 
centage  values  Increase  to  49  and  38  percent,  respectively.  Thus,  it 
could  be  anticipated  that  the  CAI  materials  to  be  produced  could  have 
a  substantial  Impact  on  total  block  performance. 

Materials  Development.  Work  within  the  Materials  Development  com- 
ponent  of  the  model  began  with  evaluation  of  the  adequacy  of  the  block 
and  lesson  tests  and  lesson  content  outlines.  As  was  previously  noted, 
problems  with  the  adequacy  of  the  block  tests  had  been  detected  during 
the  block  selection  process.  A  number  of  deficiencies  were  also  noted 
In  the  lesson  tests  and  content  outlines.  It  was  necessary,  therefore, 
to  expend  a  substantial  amount  of  effort  In  revising  the  tests  and  out¬ 
lines  In  order  to  provide  reliable  tools  for  evaluating  the  CAI 
materials  to  be  produced.  Such  extensive  revision  of  content  and  tests 
had  not  been  anticipated.  As  revisions  were  made,  they  were  submitted 
to  WM  SMSs  for  review  and  approval.  In  some  cases,  several  cycles  were 
required  before  agreement  was  reached  between  course  and  contractor 
personnel . 

Instructional  and  Individualization  strategies  were  then  defined 
for  the  six  first-pass  modules.  Since  these  lessons  had  been  selected 
on  the  basis  of  Indications  of  lack  of  student  mastery,  emphasis  was 
placed  on  achieving  mastery  rather  than  on  attempting  to  reduce  first- 
pass  study  time.  Due  to  the  specialized  nature  of  the  subject  matter, 
few  If  any  students  would  have  prior  knowledge  of  the  lesson's  content, 
so  there  was  no  appa'ent  advantage  for  Including  pretests.  In  general, 
the  Instructional  approach  adopted  within  objectives  was  gradual  pre¬ 
sentation  of  the  facts  and  concepts  to  be  learned  with  heavy  emphasis  on 
embedded  diagnostic  questions.  Students  who  Indicated  a  lack  of 
understanding  through  poor  performance  on  the  questions  were  to  be 
routed  to  further  elaboration  of  the  material  or.  In  the  case  of  more 
serious  problems,  washed  back  to  restudy.  Where  visual  aids  were  re¬ 
quired,  supplementary  picture  books  wf  *e  to  be  used  with  specific 
photographs  and  drawings  being  refere  »d  by  the  relevant  on-line 
displays. 

Since  the  content  of  all  of  the  lesson  ,vas  considered  to  be 
strongly  hierarchical  In  nature,  the  lessor  mastery  tests  were  to  be 
broken  up  Into  their  objective-specific  subscales  and  embedded  In  the 
CAI  modules  themselves.  At  the  end  of  each  objective,  the  student 
would  be  tested  over  the  content  of  that  objective  and  failure  of  an 
objective  was  to  be  treated  as  a  lesson  test  failure.  Under  these  con¬ 
ditions,  Instruction  would  be  terminated  and  the  student  would  be 
directed  to  seek  assistance  from  his  or  her  instructor.  As  strategies 
for  specific  modules  were  defined,  they  were  discussed  with  the  course 
subject  matter  specialists.  No  particular  problems  were  encountered. 

Somewhat  different  Instructional  and  Individualization  strategies 
were  defined  for  the  block  review  and  remediation  modules  which.  It  will 


be  recalled,  were  to  be  highly  modified  versions  of  the  first-pass 
modules.  The  major  difference  was  that  the  particular  objectives  to  be 
assigned  were  to  be  selected  on  the  basis  of  the  student's  performance  on 
the  lesson  tests  within  the  block  (In  the  case  of  block  review)  or  on  the 
block  test  (In  the  case  of  block  remediation).  Further,  the  modules  were 

to  Incorporate  a  number  of  pretest  questions  to  determine  specific 

problem  areas  within  objectives  and  to  branch  to  the  relevant  Instruc¬ 
tion.  Short  objective  overviews  were  to  replace  the  bulk  of  the  "main¬ 
line"  Instruction  contained  In  the  first-pass  objectives.  The  most 
Important  features  of  each  objective  were  to  be  retained  and  completely 
new  embedded  questions  and  objective-level  test  questions  would  be  re¬ 
quired. 

The  next  step  Involved  the  actual  authoring  of  the  six  first-pass 
modules.  Work  on  three  of  the  modules  began  before  the  Authoring  Editor 

was  completed  but  after  It  had  been  designed  and  module  structure 

defined.  Therefore,  this  earliest  authoring  was  done  on  paper  display 
forms.  When  a  rudimentary  form  of  the  Editor  became  available,  materials 
which  had  been  prepared  on  forms  were  Input  by  secretaries.  It  was 
thought  that  some  of  the  authors  might  wish  to  continue  using  forms  but 
an  additional  Editor  feature  became  available,  all  of  the  authors  found 
It  more  convenient  to  Input  and  format  the  materials  themselves.  All  of 
the  work  on  the  last  three  lessons  was  accomplished  In  this  way. 

As  the  first-pass  modules  were  completed,  on-line  and  printed 
copies  of  the  materials  were  submitted  to  the  course  SMSs  for  review  and 
approval.  Other  than  the  fact  that  this  process  was  relatively  time  con¬ 
suming,  relatively  few  problems  were  encountered.  For  the  most  part, 
suggested  changes  to  the  modules  were  Incorporated  as  they  were  received. 

Single-student  tryouts  were  conducted  In  parallel  with  SMS  review, 
employing  off-duty  student  volunteers  from  the  WM  course.  The  tryouts 
were  held  In  the  contractor's  on-base  facility.  The  only  serious 
problem  encountered  was  the  difficulty  In  enlisting  volunteers.  A 
computer  program  Identified  prospective  volunteers  from  average  or  above 
average  WM  students.  Those  who  chose  to  volunteer,  were  paid  $7.50  for 
2  or  3  hours  of  their  ilme.  Either  this  amount  of  remuneration  was  not 
sufficient  or  payment  In  general  may  have  no  particular  appeal;  slightly 
less  than  20  volunteers  participated. 

In  general,  the  results  of  the  tryouts  were  quite  favorable.  Some 
areas  were  found  to  be  In  need  of  Improvement  with  respect  to  clarity  and 
minor  modifications  were  made  on  a  dally  basis.  None  of  the  volunteers 
made  any  adverse  comments,  either  verbal  or  via  the  on-line  Comment 
feature,  reyarding  the  CAI  medium.  All  of  the  participants  expressed  a 
liking  for  CAI  and  thought  that  It  would  fare  well  In  the  classroom. 

Since  the  block  review  and  remediation  modules  were  highly  dependent 
on  the  first-pass  modules,  actual  authoring  of  these  modules  was  not 
begun  until  single  student  tryouts  and  subsequent  revisions  had  been 
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completed  for  the  first-pass  modules.  The  four  new  modules,  one  review 
and  one  remediation  module  for  each  of  the  two  blocks,  were  then  develop¬ 
ed  by  combining  and  modifying  copies  of  the  first-pass  modules  along  the 
lines  previously  discussed.  Since  the  first-pass  modules  had  been 
developed  for  the  most  troublesome  lessons  In  each  block.  It  could  be 
anticipated  that  on-line  materials  would  be  available  for  most  of  the  ob¬ 
jectives  assigned  for  review  or  remediation.  Objectives  for  which  on-line 
materials  were  not  available  were  simply  defined  within  the  modules  with 
directions  to  the  student  to  study  these  objectives  off-line.  As  anti¬ 
cipated,  development  of  these  modules  was  found  to  be  a  relatively  minor 
task.  As  the  modules  were  completed,  they  were  submitted  to  the  MM  SMSs 
for  review  and  approval.  Due  to  contractual  time  constraints  and  the 
experienced  difficulty  of  obtaining  student  volunteers,  the  review/ 
remediation  modules  were  not  subjected  to  single  student  tryouts. 

Formative  Evaluation  of  First-Pass  Modules.  While  single  student 
tryouts  of  the  six  first-pass  modules  were  In  progress,  preparations  were 
made  for  Implementing  the  modules  in  the  WM  course  for  purposes  of  form¬ 
ative  evaluation.  The  Block  5  and  6  Instructors  were  briefed  on  the 
purpose  and  procedures  of  the  evaluation  and  two  interactive  terminals 
were  installed  In  each  of  the  two  classrooms.  As  revisions  following 
single  student  tryouts  were  completed,  the  six  modules  were  Incrementally 
Implemented  In  the  classroom.  Each  Implementation  required  notifying 
the  Instructor  that  the  module  would  be  available  for  assignment,  pro¬ 
viding  any  necessary  supplementary  materials,  and  modifying  the  CMI  data 
base  so  as  to  assign  the  module.  Because  of  the  need  to  complete 
formative  evaluation  as  soon  as  possible,  the  assignment  rule  implemented 
was  such  that  a  student  who  was  eligible  to  study  a  lesson  supported  by 
a  CAI  module  was  assigned  the  module  If  a  terminal  was  available.  Two 
ATC  Instructors  were  on  duty  In  each  classroom  throughout  the  evaluation 
period  and  contractor  personnel  visited  the  classrooms  frequently  to 
answer  questions  and  help  resolve  operational  problems. 

Student  performance  data  were  retrieved  and  analyzed  as  they 
accumulated.  During  the  early  stages  of  the  evaluation,  only  preliminary 
versions  of  the  CAI  reports  were  available.  Therefore,  reliance  was 
placed  on  standard  reports  provided  by  the  CMI  system.  These  reports 
appeared  to  .Indicate  that  a  large  number  of  students  were  falling  the 
lesson  tests  embedded  in  the  CAI  modules  and  revisions  clearly  appeared 
to  be  called  for.  Therefore,  a  number  of  modifications  were  made, 
usually  involving  an  In-depth  treatment  of  the  more  difficult  concepts. 
These  revisions  did  not,  however,  result  in  any  substantial  Improvements 
In  apparent  student  performance.  In  the  absence  of  any  clear  solution, 
contractor  Dersonnel  were  stationed  in  the  classrooms  on  a  full-time 
basis  to  more  closely  observe  procedures. 

As  a  result  of  this  more  active  monitoring,  it  was  found  that,  in 
many  cases,  students  assigned  CAI  modules  did  not  read  their  prescrip¬ 
tions  closely  enough  to  note  that  the  assignment  was  to  a  CAI  module. 

They  studied  the  conventional  materials,  took  the  regular  off-line  test, 
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entered  the  module  number  from  their  prescription  on  the  test  form  (as 
was  required),  and  submitted  the  completed  test  form  to  a  management 
terminal.  Since  the  number  of  the  test  form  agreed  with  the  number  of 
the  module  assigned.  It  was  accepted,  and  the  student's  performance 
record  was  flagged  as  being  the  result  of  the  CAI  treatment.  In 
general.  It  tended  to  be  the  poorer  students  who  made  this  type  of  error 
and  It  was  their  data  which  were  falaclously  inflating  the  CAI  module 
failure  rates. 

Since  the  objective-level  CAI  tests  did  not  produce  a  total  test 
score  while  the  conventional,  off-line  tests  did,  this  difference  was 
used  to  discriminate  the  data  of  students  who  had  actually  studied  via 
CAI  from  the  erroneous  data.  This  more  accurate  measure  Indicated  that 
the  CAI  modules  were,  with  only  minor  exceptions,  actually  performing 
very  effectively.  While  It  would  have  been  preferable  to  remove  the 
revisions  made  on  the  basis  of  the  falaclous  data  and  begin  formative 
evaluation  anew,  such  action  was  precluded  by  the  limited  time  remaining 
In  the  contract.  As  a  result,  the  CAI  modules  were  not  as  "lean"  as 
they  might  have  been. 

To  avoid  such  problems  In  the  future,  a  change  was  made  to  the  CMI 
software  such  that  an  off-line  test  would  not  be  accepted  from  a  student 
who  had  been  assigned  a  CAI  module.  In  such  cases,  the  Instructor  was 
required  to  override  the  student's  assignment  to  the  off-line  module 
actually  studied.  Once  the  Instructors  were  made  aware  of  the  problem, 
they  also  monitored  students'  assignments  more  closely  and  the  incidence 
of  students  studying  the  wrong  module  decreased. 

The  only  other  major  problem  encountered  was  also  operational  In 
nature.  Even  when  allowance  was  made  for  students  falling  to  study  the 
CAI  modules  when  they  were  assigned,  evaluation  data  did  not  accumulate 
as  rapidly  as  had  been  anticipated.  Well  Into  the  formative  evaluation 
period,  an  assignment  rule  error  was  detected  and  corrected.  While  this 
Improved  the  situation  somewhat,  a  second  error  In  the  data  base  was 
later  found  to  also  be  depressing  the  frequency  of  CAI  assignments.  This 
second  error  was  not  discovered  until  after  summatlve  evaluation  had 
begun. 

both  students  and  Instructors  expressed  a  liking  for  CAI  as  an 
Instructional  medium,  and  they  made  no  negative  comments,  either  verbally 
or  via  the  CAI  Presentation  Program's  Comment  facility. 

Summatlve  Evaluation  Procedures*  Within  the  context  of  this  pro¬ 
ject,  the  primary  purpose  of  summatlve  evaluation  was  to  compare  the 
Instructional  effectiveness  of  the  CAI  materials  developed  with  that  of 
the  original  materials  and  procedures.  As  was  stated  In  the  Intro¬ 
duction  to  this  section,  the  evaluation  was  only  partial  In  the  sense 
that  Individualized  student  assignment  rules  had  not  been  defined  and 
validated.  Furthermore,  there  had  not  been  an  opportunity  to  submit  the 
review  and  remediation  modules  to  any  type  of  formative  evaluation. 
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These  materials  were  being  tried  out  for  the  first  time.  The  evaluation 
criteria  were  also  limited  to  the  effect  of  the  CAI  modules  on  first- 
attempt  study  times,  lesson  times  to  criterion,  lesson  test  performance, 
and  performance  on  the  end-of-block  tests  for  the  two  blocks  containing 
the  CAI  modules.  The  possible  effects  of  the  CAI  on  subsequent  lessons 
and  blocks  were  not  examined. 

Relatively  little  implementation  effort  was  required  for  this  phase. 
Instructors  were  briefed  as  to  the  purpose  of  and  procedures  for 
summatlve  evaluation  and  two  additional  Interactive  terminals  were  In¬ 
stalled  In  each  of  the  two  classrooms  to  facilitate  data  collection. 
Contractor  personnel  were  not  present  In  the  classroom  during  sumnatlve 
evaluation  but  by  then  the  Instructors  had  become  proficient  In  the  CAI- 
related  procedures  and  were  supportive  of  efforts  to  conduct  a  meaning¬ 
ful  evaluation  In  an  operational  mode. 

Summatlve  evaluation  data  collection  was  conducted  In  two  phases. 
With  relatively  little  time  remaining  In  the  contractual  period,  there 
was  a  danger  that  adequate  sample  sizes  could  not  be  obtained  for  all 
10  modules.  Since  evaluation  of  the  first-pass  modules  was  considered 
to  be  of  greater  Importance,  Implementation  of  the  block  review  and 
remediation  modules  was  postponed  until  samples  of  substantial  size  had 
been  established  for  the  first-pass  modules.  For  the  first-pass  modules, 
the  assignment  rules  were  the  same  as  had  been  employed  during  formative 
evaluation.  Thus,  assignment  was  on  a  quasi-random  basis,  l.e.,  CAI 
was  assigned  If  a  terminal  was  available.  Overall,  this  resulted  In 
approximately  25%  of  the  assignments  on  relevant  lessons  being  made  to 
CAI  modules  and  75%  to  the  original  materials.  In  addition.  It  was 
discovered  that  some  Instructors  were  overriding  students'  assignments 
to  CAI  as  they  thought  that  the  CAI  treatment  would  be  beneficial  to 
every  student.  These  Instructor  assignments  were  subsequently  excluded 
during  data  analysis. 

Once  It  became  apparent  that  adequate  samples  would  be  obtained 
for  the  first-pass  modules,  the  CMI  data  base  was  modified  to  assign 
the  CAI  block  review  and  remediation  modules.  The  same  assignment  rule 
(assign  CAI  If  a  terminal  Is  available)  was  employed  for  the  review 
modules.  Since  the  CHI  system  did  not  manage  block  remediation,  the 
block  remediation  modules  were  assigned  manually.  As  the  first-pass 
modules  achieved  sample  sizes  which  were  considered  to  be  adequate,  they 
were  withdrawn  to  provide  more  terminal  time  for  the  review  and 
remediation  modules. 

During  the  evaluation  period,  there  were  only  a  minimal  number  of 
block  test  failures.  Thus,  It  was  not  possible  to  evaluate  the 
effectiveness  of  the  CAI  block  remediation  modules. 

While  the  CAI  data  collection  routines  were  retained  In  an  active 
state,  all  of  the  data  required  for  summatlve  evaluation  were  collected 
by  the  standard  AIS  CMI  data  collection  programs.  The  data  were  then 
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retrieved  for  analysis  via  the  system's  Data  Extraction  Program  (DEP). 

Chi  Square  tests  of  frequency  data  were  done  directly  from  the  DEP  out¬ 
puts.  For  the  analyses  of  variance  of  time  data,  DEP  was  used  to  create 
a  data  file  which  was  then  submitted  to  SPSS  (Me,  Hull,  Jenkins, 
Stelnbrenner,  &  8ent,  1975)  for  analysis. 

ATC  Author  Training 

Since  a  major  design  goal  of  the  CAI  Authoring  System  was  that  ATC 
personnel  be  able  to  learn  to  use  the  System  with  very  little  formal 
training,  a  second  aspect  of  the  System  evaluation  addressed  this 
question. 

Since  the  Ideal  situation  would  have  been  for  ATC  personnel  to 
learn  to  develop  materials  while  continuing  their  normal  duties,  this 
was  the  first  approach  attempted.  Copies  of  the  Authoring  Procedures 
Handbook  were  given  to  a  number  of  volunteer  Instructors  In  the 
Weapons  Mechanic,  Inventory  Management,  and  Precision  Measuring  Equip¬ 
ment  courses  with  the  understanding  that  they  were  to  use  It,  unassisted 
by  formal  training,  to  develop  CAI  materials  for  use  In  their  class¬ 
rooms.  They  were  to  work  on  the  materials  whenever  It  would  not  Inter¬ 
fere  with  their  normal  duties.  All  of  the  Instructors  either  had  a 
terminal  In  their  classroom  or  had  relatively  convenient  access  to  a 
terml nal . 

This  approach  was  a  complete  failure  since  none  of  the  Instructors 
found  time  to  develop  materials  within  the  confines  of  their  dally 
duties.  In  some  Instances,  they  did  have  time  to  peruse  the  Handbook  and 
made  several  worthwhile  suggestions  for  changes. 

Given  that  a  somewhat  more  formal  approach  was  apparently  required, 
permission  was  sought  and  received  from  ATC  for  a  small  number  of  In¬ 
structors  to  be  relieved  of  their  duties  on  a  part-time  basis  to  attend 
a  semi-formal  CAI  development  course.  Three  Instructors,  drawn  from  the 
Weapons  Mechanic,  Inventory  Management,  and  Precision  Measuring  Equipment 
courses  participated.  None  of  the  author  trainees  were  computer  pro¬ 
grammers,  and  none  had  prior  CAI  development  experience. 

The  training  was  conducted  for  a  period  of  3  to  4  hours  each 
morning  over  a  period  of  3  weeks.  During  the  first  session,  contractor 
personnel  presented  an  overview  of  the  CAI  system,  discussed  the  role 
of  CAI  within  the  AIS,  and  provided  a  general  Introduction  to  the 
Authoring  Editor  and  the  Authoring  Procedures  Handbook.  No  formal 
training  took  place  during  the  subsequent  14  sessions.  Using  the  Hand¬ 
book  as  a  reference  manual,  each  of  the  trainees  used  the  Ai  thorlng 
Editor  to  develop  a  CAI  module  In  the  area  of  their  own  specialty.  Con¬ 
tractor  personnel  were  available  to  answer  questions  and  to  review  and 
comment  on  the  trainees'  work.  On  occasion,  the  author  trainees  were 
also  able  to  work  on  their  modules  during  the  afternoons  while  performing 
their  normal  classroom  duties. 
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V.  EVALUATION  RESULTS 


The  AIS  CAI  Authoring  System  was  evaluated  from  three  differing 
standpoints:  (a)  the  time  required  for  CAI  materials  development  by 
contractor  personnel,  (b)  the  Instructional  effectiveness  of  the  CAI 
modules  developed  by  the  contractor,  and  (c)  the  effectiveness  of  the 
CAI  development  training  for  ATC  Instructors.  The  results  of  each  of 
these  three  evaluation  efforts  are  reported  In  the  following  subsections. 

CAI  Materials  Development  Times 

The  one  aspect  of  the  contractor's  CM  materials  development  effort 
that  was  more  time  consuming  than  had  been  anticipated  concerned  the 
work  required  In  revising  the  lesson  content  outlines  and  the  eval¬ 
uation  Instruments.  Since  the  original  Instructional  materials  and  tests 
were  available.  It  had  been  thought  that  this  step  In  the  procedural 
model  would  be  relatively  trivial.  In  fact,  the  lesson  content  out¬ 
lines  required  extensive  revision,  and  It  was  necessary  to  separate 
cognitive  and  performance  elements  which  had  been  embedded  In  the  same 
lessons.  To  assure  a  reasonable  basis  for  evaluating  the  CAI  materials, 
extensive  revisions  of  the  lesson  and  block  tests  were  also  required. 

In  retrospect,  the  magnitude  of  the  problems  In  this  area  should  have 
been  anticipated  since  the  lessons  selected  for  CAI  development  re¬ 
presented  obvious  problem  areas.  All  other  aspects  of  the  development 
process  were  accomplished  at  least  as  quickly  as  had  been  hoped. 

The  contractor's  CAI  authoring  team  consisted  of  three  members.  All 
were  experienced  technical  training  authors  of  programmed  text  and 
audio-visual  materials,  but  none  had  any  prior  CAI  authoring  experience. 
In  fact,  none  had  even  used  CAI  as  a  student.  None  had  more  than  minimal 
computer  programming  experience.  Only  one  of  the  three  could  be  con¬ 
sidered  a  WM  subject  matter  expert. 

The  team  did  not  keep  accurate  records  of  development  times  but 
times  can  be  estimated  for  the  six  first-pass  modules.  At  the  end  of  the 
first  G  months  of  the  project,  the  first-pass  modules  had  been  revised 
following  formative  evaluation  and  implemented  for  suinmatlve  evaluation. 
The  team  leader  spent  relatively  little  time  actually  authoring,  con¬ 
centrating  Instead  on  producing  the  Authoring  Procedures  Handbook,  Inter¬ 
facing  with  the  Authoring  System  software  personnel,  and  attending  to 
administrative  problems.  The  other  two  team  members  were  occasionally 
called  upon  for  assistance  on  other  on-going  projects.  A  liberal 
estimate  of  the  total  time  spent  In  developing,  evaluating,  and  re¬ 
vising  the  six  first-pass  modules  Is  2200  hours.  This  Includes  the  time 
spent  In  revising  the  block  and  lesson  tests  even  though  the  former  was 
not  technically  part  of  the  CAI  effort. 

According  to  the  course's  Plan  of  Instruction  (POI),  the  content 
taught  by  the  CAI  modules  was  equivalent  to  approximately  2r>  classroom 
hours.  On  this  basis,  CAI  development  required  bo  work  hours  per  POI 
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hour.  Average  student  completion  time,  totalled  across  the  six  modules, 
was  18./  hours,  resulting  In  an  estimated  118  work  hours  per  student 
contact  hour  for  first-pass  modules.  Development  time  for  each  contact 
hour  of  review  and  remediation  materials  amounted  to  approximately  lb 
additional  hours. 

The  value  of  118  hours  per  hour  compares  quite  favorably  with  the 
contractor's  experienced  development  times  for  programmed  text  (100  work 
hours  per  student  contact  hour)  and  audio-visual  (135  work  hours  per 
contact  hour)  materials  In  the  AIS  environment.  It  also  compares 
extremely  well  with  other  CAI  authoring  systems.  In  a  study  reported  by 
Hlmwlch  (1977),  It  was  found  that  military  personnel  working  as  T ICC  I T 
and  PLATO  prog rammer/ authors  required  246  and  222  hours  of  development, 
respectively,  per  each  student  contact  hour.  Since  CAI  development 
using  the  AIS  Authoring  System  approaches  the  development  time  for  pro¬ 
grammed  text  materials  and  surpasses  that  of  audio-visual  materials, 
the  effort  required  appears  to  be  more  comparable  to  conventional  media 
development  times  than  to  the  time  requirements  of  other  CAI  authoring 
systems. 

CAI  Module  Instructional  Effectiveness 


Results  of  the  summatlve  evaluation  of  the  contractor-produced  CAI 
materials  will  be  discussed  In  two  parts:  evaluation  of  the  six  first- 
pass  modules,  and  evaluation  of  the  two  block  review  modules. 

First-Pass  Module  Summatlve  Evaluation  Results.  Evaluation  of  the 
six  CAI  modules  developed  for  first-pass  study  was  based  on  comparison 
of  these  materials  with  the  original  materials  with  respect  to  five 
student  performance  criteria:  (a)  first-attempt  lesson  test  failure 
rates,  (b)  objective-level  failures  on  the  first-attempt  lesson  tests, 

(c)  first-attempt  study  times,  (d)  lesson  times  to  criterion,  and  (e) 
objective-level  failures  on  the  first-attempt  end-of-block  tests. 

For  the  purpose  of  evaluating  first-attempt  lesson  test  failures, 
student  performance  records  were  selected  on  the  following  basis:  module 
assignment  determined  by  the  CMI  system  (l.e.,  at  random)  rather  than  by 
an  Instructor,  all  work  on  the  module  completed  during  normal  class 
hours,  and  CMI  system  checks  on  the  data  Indicating  that  they  could  be 
considered  to  be  reliable.  Performance  observations  were  then  differ¬ 
entiated  on  the  basis  of  module  number  (l.e.,  module  number  1  for  the 
conventional  materials  and  module  number  5  for  CAI). 

The  data  resulting  from  these  analyses  are  presented  In  Table  1. 

In  general,  averaging  across  the  six  lessons,  assignment  of  a  CAI  module 
did  result  in  a  substantial  reduction  In  lesson  test  failures,  from  23% 
for  the  conventional  materials  to  just  6%  for  CAI.  This  represents  a 
reduction  In  failure  rate  of  almost  75%.  Failure  rates  were  numerically 
reduced  on  all  six  lessons  although  the  differences  (as  tested  by  Chi 
Square)  were  only  significant  In  three  of  the  six  cases. 
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Table  1.  First-Attempt  Lesson  Failure  Rates  as  a  Function  of 

Module  Type 


Block 

Number 

Lesson 

Number 

Conventional 

CAI 

— 

n 

Percent 

Failure 

n 

Percent 

Failure 

2 

206 

19 

31 

06 

3.11 

5 

3 

70 

14 

64 

00 

9.88  ** 

4 

105 

25 

63 

02 

15.68  *** 

7 

191 

37 

30 

03 

13.20  *** 

1 

83 

24 

83 

17 

1.33 

6 

6 

74 

19 

55 

09 

2.43 

Average  Over 

All  Lessons 

23 

06 

■  ■ 

It  should  be  noted  that,  given  the  testing  strategy  employed  In 
the  CAI  modules,  there  were  more  opportunities  to  fall  the  lesson  test 
than  was  the  case  for  the  conventional  modules.  That  Is,  a  student 
assigned  a  CAI  module  was  considered  to  have  failed  the  lesson  following 
the  first  objective  failure.  For  the  conventional  modules,  the  student 
was  only  required  to  meet  an  overall  test  criterion  based  on  the  percent¬ 
age  of  Items  answered  correctly. 

With  the  exception  of  Lesson  1  In  Block  6,  the  CAI  module  failure 
rates  were  considered  to  be  satisfactory,  i.e.,  less  than  10*.  This 
module's  17*  failure  rate  Indicated  a  need  for  further  revision. 

A  more  detailed  evaluation  of  lesson  test  performance,  at  the 
objective  level.  Is  presented  In  Table  2.  The  selection  procedures 
employed  in  these  analyses  were  the  same  as  those  used  In  the  compar¬ 
ison  of  overall  lesson  failure  rates.  Hence,  the  sample  sizes  shown 
for  the  first  objective  In  each  lesson  are  the  same  as  for  Table  1.  For 
the  CAI  modules,  the  number  of  observations  on  subsequent  objectives  are 
reduced  due  to  students  falling  an  objective  and  exiting  the  lesson. 

As  Indicated  in  Table  2,  the  effect  of  tie  CAI  modules  was  more 
pronounced  at  the  objective  level.  Overall,  averaging  across  the  21 
objectives,  assignment  of  a  CAI  module  reduced  the  objective  failure 
rate  from  19%  to  only  2%,  and  89%  reduction  In  the  proportion  of 
objective  failures.  The  CAI  failure  percentages  were  numerically  less 
than  those  of  the  conventional  modules  on  all  21  objectives  and  the 
differences  were  statistically  significant  In  16  of  the  21  cases. 

There  was  only  one  Instance  In  which  CAI  failures  exceeded  5%  -- 
Objective  2  In  Block  6,  Lesson  1.  It  can  be  seen  that  the  high  lesson- 
level  failure  rate  for  this  module  can  be  attributed  to  a  number  of 
relatively  high  objective  failure  rates  and  the  rule  that  the  lesson  be 
considered  to  be  failed  as  soon  an  an  objective  was  failed.  On  the 
other  hand,  it  should  also  be  noted  that  this  module  was  found  to  be 
one  of  the  most  effective,  relative  to  the  conventional  materials.  In 
that  three  of  the  six  objective  failure  rate  differences  were  statis¬ 
tically  significant  at  the  .001  level. 

It  will  be  recalled  that  the  major  design  goal  of  the  first-pass 
CAI  modules  was  to  increase  mastery  (to  reduce  the  number  of  first- 
attempt  lesson  and  block  test  failures)  rather  than  to  reduce  study 
time.  It  was  still  of  Interest  to  contrast  the  study  times  of  the  CAI 
modules  with  those  of  the  conventional  materials. 

A  number  of  additional  restrictions  pertaining  to  the  reliability 
of  the  measured  first-attempt  study  times  were  added  to  the  sample 
selection  criteria.  Due  to  time  measurement  problems  raised  by  certain 
classroom  procedures  (e.g.,  periods  during  the  normal  classroom  day 
devoted  to  administrative  work)  and  the  CHI  system's  approach  to  re¬ 
solving  these  problems,  a  relatively  small  proportion  of  the  students 
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Table  2.  First-Attempt  Lesson  Objective  Failure  Rates  as  a 


Function  of  Module  Type 


were  considered  to  have  reliable  first-attempt  study  times. 

The  results  of  the  analyses  are  presented  In  Table  3.  With  only 
two  exceptions,  first-attempt  study  times  were  roughly  equivalent  for 
the  two  module  types.  For  Lesson  1  In  Block  6,  the  CAI  module  required 
only  60%  of  the  time  required  by  the  conventional  materials.  For  Lesson 
2  In  Block  5,  on  the  other  hand,  the  CAI  module  required  118*  of  the 
average  time  required  by  the  conventional  module.  It  will  be  recalled 
that  the  CAI  modules  had  been  revised  and  expanded  on  the  basis  of 
falaclous  student  performance  data  during  formative  evaluation.  The 
Block  5,  Lesson  6  module  was  substantially  expanded  at  that  time. 

Overall,  across  the  six  modules,  cumulative  first-attempt  study 
time  was  reduced  from  1179  to  1121  minutes,  a  reduction  of  4.9%. 

Although  It  Is  not  possible  to  estimate  how  much  less  time  the  CAI 
modules  might  have  required  If  unnecessary  revisions  had  not  been  made. 

It  Is  likely  that  the  difference  would  have  been  substantial  since 
relatively  large  amounts  of  elaboration  were  added  to  some  of  the 
modules. 

A  more  pertinent  time  measure,  which  encompasses  both  module  study 
time  and  lesson  failure  rate,  Is  total  time  to  lesson  criterion.  The 
restrictions  required  to  assure  reliable  tlmes-to-criterlon  were  not  as 
extensive  as  those  required  for  first-attempt  module  times.  Hence, 
sample  sizes  were  not  as  seriously  diminished.  The  results  of  the  tlme- 
to-crlterlon  analyses  are  presented  In  Table  4. 

While  there  was  only  one  Instance  In  which  a  CAI  module  required 
significantly  less  time  to  criterion  than  the  corresponding  conventional 
module,  the  CAI  module  times  were  numerically  less  In  five  of  the  six 
cases.  Overall »  across  the  six  lessons,  the  cumulative  time  required 
for  the  CAI  modules  was  154  minutes  or  11.2%  less  than  that  required  for 
the  conventional  materials. 

The  final  evaluation  criterion  concerned  the  effect  of  the  first- 
attempt  CAI  modules  on  subsequent  retention  as  measured  by  objective 
failures  on  the  end-of-block  tests.  The  data  selection  criteria 
employed  were  the  same  as  those  used  In  evaluation  of  the  lesson  ob¬ 
jective  failure  rates  with  the  additional  constraint  that  the  students 
to  be  selected  had  not  also  been  assigned  the  CAI  block  review  module. 

The  results  of  these  analyses  are  presented  In  Table  5. 

While  the  differences  between  the  two  module  types  were  significant 
In  only  four  cases,  the  objective  failure  rates  for  the  CAI  modules  were 
numerically  less  than  those  of  the  conventional  modules  in  16  of  the  21 
cases.  Overall,  across  all  21  objectives,  the  block  test  objective 
failure  rate  was  only  10%  for  students  who  received  CAI  as  opposed  to  16% 
for  students  assigned  the  conventional  materials  on  their  first  attempt 
on  the  lessons. 
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Table  3.  First-Attempt  Lesson  Times  as  a  Function  of  Module  Type 


Table  4.  Lesson  Times  to  Criterion  as  a  function  of  Module  Type 
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Table  5.  First-Attempt  Block  Test  Objective  Failure  Rates  as  a 
Function  of  First-Attempt  Module  Type 


To  summarize.  In  comparison  to  the  conventional  materials,  the  six 
first-pass  modules  produced  via  the  CAI  Authoring  System  (a)  reduced 
average  lesson  failure  rates  from  23 %  to  6%,  (b)  reduced  average  lesson 
objective  failure  rates  from  19%  to  2%,  (c)  reduced  cumulative  first- 
attempt  study  time  by  4.9%,  (d)  reduced  cumulative  lesson  time  to 
criterion  by  11.2%,  and  (e)  reduced  the  average  end-of-block  test 
objective  failure  rate  from  16%  to  10%. 

Review  Module  Summatlve  Evaluation  Results.  Due  to  their  nature, 
summatlve  evaluation  of  the  two  block  review  modules  was  less  extensive 
than  was  the  case  for  the  six  first-pass  modules.  Block  test  objective 
failure  rates  were  examined  as  a  function  of  review  module  type.  The 
data  selection  criteria  employed  were  that  the  student  was  assigned  the 
particular  module  at  random  by  the  CMI  system,  that  no  part  of  the  re¬ 
view  lesson  was  completed  as  homework,  that  the  data  were  considered 
reliable  by  the  system,  and  that  the  student  had  failed  the  objective  In 
question  during  his  or  her  first-pass  study  In  the  block.  This  latter 
criterion  was  the  basis  on  which  objectives  were  recommended  for  study 
under  the  conventional  block  review  procedures  and  presented  for  study 
within  the  CAI  review  modules.  The  results  of  the  analyses  are  pre¬ 
sented  In  Table  6. 

Since  only  data  from  those  students  who  had  failed  the  objective  In 
question  during  their  first-pass  study  were  considered,  the  sample  sizes 
for  each  objective  were  quite  small.  None  of  the  Chi  Square  tests  made 
were  significant.  In  fact,  only  three  of  the  Chi  Square  values  exceeded 
1.0.  The  results  were  still  considered  to  be  encouraging,  however,  since 
over  the  total  of  492  observations  for  the  conventional  review  pro¬ 
cedures  and  153  observations  for  CAI,  the  average  objective  failure  rate 
was  reduced  from  15%  to  11%  by  assignment  of  CAI  review. 

ATC  Author  Training 

It  will  be  recalled  that  the  first  approach  to  training  ATC 
personnel  In  CAI  development  attempted  to  have  Instructors  learn  to  use 
the  Authoring  System  while  continuing  their  normal  classroom  duties. 

This  was  not  at  all  successful;  none  of  the  Instructors  found  time  for 
the  extra  work. 

As  a  means  of  providing  the  author  trainees  with  time  for  CaI 
development  work,  the  second  approach  to  training  employed  a  training 
course  which,  although  Informal,  did  have  a  scheduled  meeting  time. 

Three  ATC  Instructors,  one  from  each  of  three  courses  supported  by  the 
AIS,  came  to  the  contractor's  facility  for  3  to  4  hours  each  morning, 

5  days  a  week  for  3  weeks.  No  formal  training  took  place  after  the 
first  session  but  contractor  personnel  were  available  to  answer  questions 
and  to  review  and  comment  on  the  trainees'  work.  On  occasion,  the 
trainees  were  able  to  continue  their  CAI  authoring  work  during  their 
normal  afternoon  classroom  duties. 
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Table  6.  First-Attempt  Block  Test  Objective  Failure  Rates  as  a 
Function  of  Review  Module  Type 


Block 

Number 

Objective3 

Number 

Conventional  | 

CAI  | 

Chi 

Square 

n 

Percent 

Failure 

n 

Percent 

Failure 

2-1 

21 

10 

12 

00 

1.22 

2-2 

35 

14 

14 

14 

<1.0 

2-3 

38 

34 

13 

31 

<1.0 

3-1 

6 

17 

3 

00 

<1.0 

5 

3-2 

10 

00 

6 

17 

1.78 

3-3 

12 

08 

5 

00 

<1.0 

4-1 

38 

03 

8 

00 

<1.0 

4-2 

15 

00 

3 

00 

0.0 

7-1 

13 

00 

1 

00 

0.0 

7-2 

16 

37 

D 

23 

<1.0 

7-3 

48 

25 

m 

16 

<1.0 

1-1 

4 

24 

50 

<1.0 

1-2 

41 

37 

40 
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1-3 

22 

45 

25 

<1.0 

1-4 

25 

12 

6 

00 

<1.0 

1-5 

24 

00 

8 

00 

0.0 

6 

6-1 

13 

00 

2 

00 

0.0 

6-2 

17 

12 

6 

00 

<1.0 

6-3 

24 

04 

5 

00 

<1.0 

6-4 

24 

21 

5 

00 

1.26 

6-5 

46 

17 

7 

14 

<1.0 

Average  Over 

All  objectives 

15 

11 

a  Corresponds  to  Lesson  Number-Objective  Number  as  taught  In  the  block 


At  the  end  of  the  3-week  period,  each  author  trainee  had  developed 
a  module,  had  It  reviewed  by  the  contractor  and  other  ATC  personnel, 
had  run  limited  single-student  tryouts,  and  had  made  minor  revisions  on 
the  basis  of  these  reviews  and  tryouts.  Due  to  the  time  lost  during  the 
first,  unsuccessful  training  attempt,  the  training  came  too  late  In  the 
contract  period  to  allow  for  material  Implementation  and  evaluation  In 
the  classrooms.  One  of  the  modules  was  Implemented  briefly  In  the  WM 
course  but  no  data  were  collected  since  a  course  reorganization  which 
occurred  shortly  threrafter  eliminated  the  lesson.  From  what  Is  known 
through  observation  and  verbal  comments,  the  module  fared  well  during 
Its  short  period  of  use. 

In  the  opinion  of  the  contractor  personnel  monitoring  the  training, 
the  author  trainees  became  reasonably  proficient  at  using  the  Authoring 
Editor.  The  trainees  asked  relatively  few  questions  after  the  first  few 
sessions.  Most  of  the  suggestions  made  by  contractor  personnel  per¬ 
tained  to  the  need  for  more  frequent  questions  In  the  modules  and  In¬ 
creased  Individualization  through  branching.  The  three  modules  developed 
all  contained  branching  logic,  feedback,  prompts,  and  some  simple 
graphics.  The  consensus  of  those  reviewing  the  modules  was  that  they 
were  of  good  quality  and  had  capitalized  well  on  the  capabilities  of  CAI. 
In  general,  this  learning  was  accomplished  almost  solely  through  use  of 
the  Handbook  and  the  guidance  provided  by  the  Editor  itself.  A  very 
small  portion  of  the  training  time  was  taken  up  by  the  trainees  seeking 
assistance  or  being  given  guidance. 

The  comnents  of  the  author  trainees  were  all  favorable  and  their 
enthusiasm  for  the  project  was  very  high.  All  of  the  trainees  expressed 
an  Interest  In  having  CAI  Implemented  In  their  respective  courses. 

The  time  needed  to  train  the  ATC  Instructors  to  use  the  AIS  CAI 
Authoring  System  was  roughly  equivalent  to  8  full  days.  This  compares 
very  well  with  Hlmwlch's  (1977)  report  of  7  weeks  of  familiarization 
and  3  weeks  of  training  for  T ICC  IT  authors,  and  2  weeks  of  training  and 
subsequent  consulting  support  from  PLATO  project  personnel  for  PLATO 
authors. 

The  author  trainees  kept  records  of  the  time  spent  In  module  develop¬ 
ment.  On  the  average,  the  time  required  to  produce  a  module  ready  for 
formative  evaluation  was  90  hours  per  student  contact  hour.  While 
additional  time  would  have  been  required  for  formative  evaluation  and 
revision.  It  appears  that  the  total  time  would  have  approximated  the 
estimated  development  times  of  contractor  personnel. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 

The  authoring  system  approach  described  here  appears  promising. 
Experience  to  date  has  demonstrated  that  reasonably  effective  CAI  can  be 
produced  at  a  very  acceptable  cost  In  terms  of  work-hours  per  student 
contact  hour  and  that  ATC  personnel  can  learn  to  use  the  system  in  a 
reasonable  period  of  time  without  formal  training.  As  It  stands,  the 
authoring  system  Is  ready  for  use  by  ATC  Instructional  development 
and  evaluation  personnel.  This  final  section  summarizes  the  observed 
strengths  and  weaknesses  of  the  authoring  system,  suggests  specific  areas 
for  further  development,  and  outlines  a  number  of  recommendations  for 
system  utilization. 

Effectiveness  of  the  Authoring  System  and  Suggestions  for  Further 
Deve‘1  opment 

On  the  basis  of  the  performance  results  discussed  In  the  preceding 
section  of  this  report.  It  can  be  concluded  that  the  CAI  materials  pro¬ 
duced  under  this  effort  were  reasonably  effective;  substantially  more 
than  the  lesson  materials  which  they  replaced.  Assignment  of 
the  six  first-pass  modules  resulted  In  dramatic  reductions  In  objective 
failure  rates  on  the  lesson  tests  and  substantive,  although  not  as  strik¬ 
ing,  Improvements  In  retention  as  measured  by  objective  failures  on  the 
end-of-block  tests. 

It  must  be  admitted  that.  In  terms  of  one  of  the  more  central 
measures  of  success,  total  time  to  criterion,  the  11.2%  reduction  demon¬ 
strated  Is  not  as  dramatic  as  might  have  been  hoped  for.  These  savings, 
however,  must  also  be  evaluated  In  the  light  of  the  CAI  materials  de¬ 
velopment  effort  required.  It  will  be  recalled  that,  on  the  average, 
the  six  first-pass  modules  required  only  118  work  hours  of  development 
time  per  student  contact  hour,  a  labor  requirement  which  Is  comoarable 
to  production  of  conventional  programmed  text  and  audio/visual  materials 
and  much  less  than  the  typical  time  requirements  of  CAI  development  with 
conventional  CAI  authoring  systems.  From  this  standpoint,  the  product 
of  the  CAI  authoring  system  reported  here  appears  to  be  extremely  cost 
effective. 

Each  of  the  three  major  components  of  the  system,  the  procedural 
model,  the  support  software,  and  the  Authoring  Handbook  will  now  be 
examined  and  discussed  In  some  detail. 

Procedural  Model.  Overall,  the  evaluation  results  appear  to  support 
the  developmental  approach  outlined  by  the  procedural  model  for  the 
particular  application  for  which  It  was  designed.  While  problems  were 
encountered  In  two  areas,  they  were  more  of  the  nature  of  management 
problems  rather  than  problems  with  the  model  per  se. 

First,  Insufficient  effort  was  originally  scheduled  for  the  tasks 
of  evaluating  and  revising  the  criterion  measurement  Instruments  and 
the  content  outlines  for  the  lessons  selected  for  CAI  development. 
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Given  that  these  lessons  represented  known  problem  areas,  planning  for 
materials  development  should  have  made  greater  allowance  for  at  least 
a  portion  of  these  problems  deriving  from  inadequacies  of  content 
definition  and  Inaccuracies  In  the  criterion  tests. 

The  second  problem  area  concerned  Inadequate  emphasis  being  placed  on 
close  supervision  of  procedures  during  formative  and  summative  evaluation. 
This  was  found  to  be  particularly  critical  when  the  CAI  medium  Itself  was 
being  Introduced  Into  a  course  for  the  first  time.  During  the  eval¬ 
uation  effort,  problems  resulting  from  a  lack  of  closer  supervision  arose 
In  both  classroom  operations  and  In  definition  of  management  rules  In  the 
data  base. 

During  formative  evaluation,  the  problem  of  student  performance 
data  becoming  contaminated  by  the  data  of  students  who  were  assigned  but 
did  not  study  the  CAI  modules  could  have  been  avoided  If  more  complete 
checks  had  been  provided  by  the  CMI  system.  Past  experience  with  the 
operational  AIS,  however,  has  shown  that  It  Is  next  to  Impossible  to 
anticipate  the  variety  of  ways  In  which  assumed  procedures  will  be 
violated,  whether  Intentionally  or  Inadvertently.  A  more  practical 
solution  to  problems  of  this  type  Is  to  provide  much  closer  supervision 
of  classroom  procedures,  at  least  during  the  Initial  stages  of  Imple¬ 
mentation. 

Another  problem  In  this  area  concerned  errors  In  the  module  assign¬ 
ment  loqlc.  While  the  AIS  assignment  rules  are  not  unduly  complex,  they 
were  designed  to  allow  for  a  wide  variety  of  approaches  to  assignment 
decisions.  Since  this  was  the  first  Instance  In  which  CAI  had  been 
Implemented  In  one  of  the  AIS  courses  on  any  major  scale,  more  emphasis 
should  have  been  placed  on  assuring  that  the  assignment  rules  were 
operating  In  the  manner  Intended. 

If  there  was  any  one  aspect  of  the  model  which  was  found  to  be 
particularly  helpful.  It  was  the  emphasis  on  constant  communication  among 
the  materials  authors,  the  course’s  subject  matter  experts,  and  the  In¬ 
structors.  While  this  communication  was  expensive  In  terms  of  both 
work-hours  and  elapsed  time.  It  was  shown  to  be  well  worth  the  effort. 
Although  there  were  occasional  problems  resulting  from  communication 
failures,  materials  review.  Implementation,  and  changes  to  classroom 
procedures,  all  proceeded  much  more  smoothly  than  had  been  the  case  for 
several  similar  prior  experiments.  At  the  conclusion  of  the  evaluation 
period,  the  course  personnel  who  had  been  Involved  remained  strongly 
supportive  of  CAI  In  general  and  of  this  specific  project  In  particular. 

There  Is  no  doubt  that  the  procedural  model  can  be,  and  hopefully 
will  be,  further  refined  through  subsequent  CAI  development  experience 
with  the  AIS  authoring  system. 

Authoring  System  Support  Software.  In  assessing  the  various  soft¬ 
ware  features  and  components  of  the  CAI  authoring  system,  the  major  con- 
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trlbutor  to  simplifying  the  task  and  hence  reducing  costs  Is  probably 
elimination  of  any  need  for  the  author  to  work  In  a  computer  language. 

All  of  the  programming  work  has  been  done  beforehand  and  provided  In 
the  form  of  the  Editor  and  Presentation  Programs.  Further  needs  for 
programing  effort  will  depend  on  how  adequately  this  software  meets  the 
requirements  of  future  applications.  Due  to  Its  flexibility.  It  Is 
anticipated  that  the  existing  software  could  serve  the  needs  of  the  A1S 
environment  for  some  time  to  come.  Eventually,  however,  developing 
author  expertise  should  justify  Increased  software  capability. 

The  second  greatest  factor  in  facilitating  the  author's  task  Is 
probably  the  extent  to  which  that  task  Is  structured  by  the  Editor.  The 
overall  structure  of  the  module  Is  determined  for  the  author,  units 
within  this  structure  are  matched  to  the  requirements  of  the  environment, 
and  the  occurrence  of  critical  units  Is  either  forced  or  prompted.  While 
the  author  retains  a  great  deal  of  flexibility,  this  flexibility  Is 
exercised  through  selection  of  specific  options  which  provide  a  degree 
of  control  over  the  authoring  process,  while  reminding  the  author  of  the 
various  courses  of  action  which  may  be  taken. 

A  third  major  factor  In  facilitating  authoring  Is  undoubtedly  the 
human-engineered,  computer-aided  Input,  formatting,  and  editing  capa¬ 
bility  provided  by  the  Editor.  Other  than  the  approach  to  defining 
branching  logic,  there  is  little  here  that  Is  novel,  only  the  application 
of  existing  technology  to  a  particular  problem  area.  Given  the  diminish¬ 
ing  cost  of  computer  use,  there  is  little  reason  not  to  provide  authors 
with  the  benefits  of  this  technology. 

It  Is  difficult,  at  this  time,  to  evaluate  the  utility  of  the 
automatic,  structured  student  performance  data  collection  and  analysis 
routines.  The  capability  provided  appeared  to  be  adequate  for  formative 
evaluation  of  the  six  first-pass  modules  but  more  extensive  prompting 
and  guidance  In  the  use  of  the  data  collection  routines  and  Interpre¬ 
tation  of  the  reports  might  be  desirable. 

There  are  certainly  recognized  inadequacies  In  the  authoring  system 
software,  primarily  In  the  areas  of  constructed  response  recognition  and 
author-generated  graphics.  It  would  be  desirable  to  develop  a  dialog 
within  the  Editor  which  would  guide  the  author  through  the  steps  of 
defining  more  sophisticated  rules  by  which  anticipated  responses  to  con¬ 
structed  response  questions  are  to  be  judged.  It  Is  also  suggested  that 
a  graphics  editor  be  developed  which  would  allow  the  non-programming 
author  to  generate  drawings  either  by  defining  basic  geometric  elements 
(e.g.,  straight  lines,  circles,  arcs)  or  through  the  use  of  a  light  pen 
or  digitizing  device.  Initial  work  In  both  of  these  areas  has.  In  fact, 
already  begun. 

In  a  totally  different  area.  It  Is  suggested  that  the  utility  of 
the  authoring  system  could  be  substantially  Increased  through  provision 
of  additional  software  tools  for  managing  the  authoring  process.  The 
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approach  envisioned  Includes  capturing  relevant  parameters  of  the 
development  process,  such  as  development  status  and  rate  of  completion, 
and  providing  managers  with  access  to  this  Information  through  summary 
displays  and  reports. 

Authoring  Procedures  Handbook.  On  the  basis  of  the  success  of  the 
limited  author  training  for  ATC  personnel.  It  Is  tentatively  concluded 
that  the  AIS  CAI  Authoring  Procedures  Handbook  did  meet  Its  goal  of  pro¬ 
viding  a  document  which  could  be  used  by  the  average  ATC  materials 
developer  to  learn  to  produce  CAI  materials  without  further  formal  train¬ 
ing.  As  Is  the  case  for  the  other  components  of  the  authoring  system, 
however,  there  Is  certainly  still  room  for  Improvement.  In  particular, 
the  Handbook  would  benefit  from  Increased  emphasis  on  the  concepts  of 
Individualization  and  the  formative  evaluation/revision  process.  More 
concrete  examples  of  problems  and  problem  solving  procedures  In  these 
areas  would  also  be  quite  useful. 

Given  that  the  Handbook  content  Is  available  on-line,  an  obvious 
step  for  further  development  would  be  to  generate  a  version  of  the  Hand¬ 
book  which  Is  more  Interactive;  a  self-instructional  program  containing 
more  practice  examples  and  criterion  tests.  Further,  the  Handbook's 
utility  as  a  reference  manual  could  be  Increased  substantially  through 
a  system  of  cross  referencing  to  specific  problem  areas.  Initial  steps 
In  this  direction  have  already  been  taken. 

Recommendations  for  CAI  System  Utilization 

The  major  conclusions  drawn  from  this  study  pertain  to  the  finding 
that  CAI  materials  can  be  developed  at  a  reasonable  cost  rather  than  to 
the  effectiveness  of  the  CAI  medium  per  se.  Thus,  the  Intent  of  the 
recommendations  outlined  below  Is  to  suggest  potential  benefits  to  be 
derived  from  broader  application  of  an  Instructional  technology  which 
has  already  been  shown  to  be  effective  In  a  variety  of  contexts. 

An  obvious  next  step  would  Involve  further  development  and  Im¬ 
plementation  of  CAI  modules  In  the  courses  supported  by  the  AIS.  While 
It  Is  suggested  that  specific  lessons  be  selected  on  the  basis  of  the 
procedures  outlined  In  the  Procedural  Model,  there  are  some  areas  of 
Instruction  In  the  current  AIS  courses  which  appear  particularly  promis¬ 
ing.  These  Include  remedial  mathematics  review  and  drill  and  practice 
In  the  Precision  Measuring  Equipment  course,  troubleshooting  concepts 
In  Precision  Measuring  Equipment  and  the  Weapons  Mechanic  course,  and  a 
variety  of  procedural  tasks  In  the  Inventory  Management  course. 

With  some  modification  to  the  presentation  program,  the  AIS  CAI 
capability  could  be  extended  to  resident  training  courses  which  are  not 
currently  supported  by  the  AIS.  For  example,  there  are  a  number  of 
promising  applications  In  courses  taught  by  Lowry's  Armed  Forces  Air 
Intelligence  Training  Center. 
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Eventually,  one  of  the  most  promising  areas  for  low  cost  CM  would 
appear  to  be  In  detached  or  on-the-job  training.  Given  the  availability 
of  terminals  located  on  operational  command  bases,  CAI  would  provide  a 
means  by  which  development  and  maintenance  of  Instructional  materials 
could  be  centralized  and  new  and  revised  materials  could  be  rapidly  dis- 
emlnated  to  the  field.  Once  such  an  Instructional  distribution  system  was 
In  place,  It  could  also  be  beneficially  employed  In  related  areas.  For 
example,  technical  orders  and  other  forms  of  documentation  maintained 
on-line  could  be  quickly  updated  at  a  central  location  and  dispatched 
Immediately  to  all  applicable  sites. 

One  aspect  of  the  AIS  CAI  authoring  system  which  was  addressed  only 
scantily  by  this  study  Is  the  availability  of  printed  copies  of  CAI 
modules.  An  obvious  application  for  such  materials  Is  as  backups  for 
CAI  modules,  to  be  used  In  case  of  computer  failure  or  an  Inadequate 
number  of  terminals.  In  addition,  however,  there  would  appear  to  be 
several  advantages  to  on-line  development  of  materials  Intended  only  for 
off-line  use.  Initial  development  would  benefit  from  the  structure  and 
guidance  provided  by  the  Editor  as  well  as  Its  text  formatting  and 
editing  capabilities.  Printed  copies  could  be  produced  as  required  for 
peer  review  or,  where  terminals  were  available,  review  could  be  on-line; 

In  which  case  the  system's  Comment  facility  would  be  available.  Re¬ 
visions  found  to  be  desirable  during  formative  evaluation  could  be  made 
Immediately  and  relevant  portions  of  the  text  reprinted  for  use  the  next 
day.  Alternatively,  formative  evaluation  Itself  could  be  conducted  on¬ 
line,  providing  more  detailed  student  performance  data  than  would  other¬ 
wise  be  available.  Once  the  materials  have  been  approved  for  final  Im¬ 
plementation,  a  copy  could  be  printed  for  reproduction  or,  if  only  a 
small  number  of  copies  are  required,  reproduction  could  be  bypassed  by 
printing  multiple  copies.  If  the  materials  are  then  archived  and  stored 
on  tape,  they  could  readily  be  retrieved,  revised,  and  reprinted  when 
subsequent  revisions  were  found  to  be  necessary. 

Any  further  Implementation  of  the  AIS  CAI  authoring  system  is  largely 
dependent  on  provision  of  opportunities  for  author  training.  In  con¬ 
gruence  with  the  Intent  of  this  project,  the  amount  of  fornal  training 
required  has  been  substantial ly  reduced  but,  as  was  discussed,  the  anti¬ 
cipation  that  training  could  take  place  on-the-job  was  apparently  un¬ 
realistic.  At  a  minimum,  It  appears  necessary  to  provide  a  designated 
meeting  time  and  place  for  the  CAI  author  training  "course."  Ideally, 
author  training  could  be  provided  as  part  of  the  normal  ATC  Instructor 
Training  Course. 
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APPENDIX  A 


GLOSSARY  OF  AIS  TERMS 


ADAPTIVE  MODEL  -  A  set  of  computer  programs  that  generate  Individual 
Instructional  assignments,  predict  and  assign  Individual  block  and 
course  completion  time  targets  (Student  Progress  Management), 
allocate  tralnlnq  resources,  and  are  the  vehicle  for  accomplishing 
continual  evaluation  and  refinement  of  courseware. 

ADVANCED  INSTRUCTIONAL  SYSTEM  (AIS)  -  A  prototype,  comprehensive  computer- 
managed  and  computer-assisted  Instructional  system  to  provide  the 
following  automated  capabilities  In  support  of  large  scale  train¬ 
ing:  Individual  instructional  assignments,  student  progress  manage¬ 
ment,  resource  allocation  and  scheduling,  Information  storage  and 
report  generation,  evaluation,  and  research  control. 

ALTERNATIVE  MODULES  -  Modules  utilizing  different  instructional  approaches 
from  previously  existing  modules  to  meet  the  specific  needs  of 
particular  types  of  students  and/or  certain  course  requi rements. 

(See  Instructional  Module). 

BLOCK  -  A  course  component  comprised  of  lessons  and  modules  that  cover  a 
specific  subject/ content  area  and  normally  ends  with  a  comprehensive 
test. 

COMPUTER-ASSISTED  INSTRUCTION  (CAI)  -  An  Instructional  mode  which  pro¬ 
vides  Instruction  via  an  interactive  computer  terminal.  Tutorial 
Instruction,  drill  and  practice,  and/or  simulation  take  place  at 
the  terminal  and  other  instructional  media  may  be  integrated  with 
the  terminal  dlsDlay.  Utilization  ratio  Is  one  student  per  inter¬ 
active  terminal. 

COMPUTER-BASED  -  CAI  and/or  CMI  support  of  Instructional  programs. 

COMPUTER  HARDWARE  -  The  hardware  components  that  comprise  the  computer 
system  and  Include  the  central  processor;  various  types  of  memory 
units;  printout  unit;  and  control,  display,  and  distribution  units. 

COMPUTER-MANAGED  INSTRUCTION  (CMI)  -  Use  of  the  computer  to  manage  stu¬ 
dents  through  the  Instructional  process.  The  computer's  role  Is 
that  of  a  diagnostician  and  manager  of  Instructional  events. 

Through  the  Adaptive  Model,  the  AIS  generates  individual  Instruc¬ 
tional  assignments,  predicts  and  assigns  Individual  block  and 
course  completion  times,  allocates  training  resources,  and 
evaluates  tests  and  courseware.  The  utilization  ratio  (In  the 
case  of  the  AIS)  is  100  students  per  management  terminal. 
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GLOSSARY  OF  TERMS  (Continued) 


COMPUTER  SOFTWARE  -  A  logical  grouping  of  programmed  computer  codes  that 
qlve  conmands  to  a  computer  to  perform  a  particular  function.  A 
unique  AIS  software  component  Is  the  Computer  Assisted/Managed 
Instructional  Language  (CAMIL)  that  facilitates  development  and 
delivery  of  both  CAI  and  CMI. 

COURSE  -  A  block  or  a  series  of  blocks  of  Instruction  designed  to  satisfy 
Specialty  Training  Standards  for  a  particular  Air  Force  Specialty 
Code  and  skill  level.  Formal,  resident  training  conducted  at  an 
Air  Training  Command  Installation. 

COURSE  DATA  BASE  -  A  collection  of  computer  files  containing  the 

parameters  and  flags  which  control  the  operation  of  the  Adaptive 
Model  for  a  specific  course. 

COURSE  EVALUATION  SUMMARY  (CES)  -  A  computer-generated  report  of  student 
performance  data  (e.g.,  average  completion  times,  scores,  and 
failure  rates)  available  for  one,  selected  combinations,  or  all 
lessons  and/or  blocks  of  a  specific  course.  The  CES  summarizes 
student  performance  at  the  module,  lesson,  and  block  level. 

COURSEWARE  -  Generic  term  for  all  AIS  instructional  material. 

HEURISTIC  MODELS  -  Logical  "If... then"  statements  used  to  assign  students 
to  alternative  .todules.  These  logical  statements  can  be  based 
solely  on  expert  judgement,  on  previously  collected  and  analyzed 
data,  or  on  a  combination  of  both. 

INSTRUCTIONAL  MATERIALS  -  Printed,  audio,  visual,  or  electronically  dis¬ 
played  Information  used  In  Instruction.  Includes  programmed  texts, 
picture  books,  workbooks,  audio/visuals,  checklists,  technical 
orders,  CAI  presentations,  and  tests. 

INSTRUCTIONAL  MODULE  -  A  specific  package  of  Instructional  materials  and 
related  training  resources  for  presentation  of  a  specific  AIS 
lesson.  A  lesson  may  have  more  than  one  Instructional  module.  All 
modules  for  a  lesson  teach  the  same  objectives  but  differ  In  the 
method  of  presentation  and/or  strategies  used. 

INTERACTIVE  (A)  TERMINAL  -  Consists  of  a  plasma  display  and  keyboard 
and  Is  used  by  Instructors  and  course  authors  to  Interact  with 
the  AIS  central  computer  and  data  files  and  Dy  students  for  on¬ 
line,  adaptive  testing  and  CAI. 

LESSON  -  The  smallest  testable  unit  of  instruction  within  a  block  of 
Instruction:  Supported  by  one  or  more  specific  Instructional 
modules  to  enable  achievement  of  a  learning  objective  or  series  of 
objectives. 


GLOSSARY  Of  It RMS  (Continued) 


MOtHILI.  -  See  Instructional  Module. 

I’Rl  ASSI  SSMLNT  bATTt.RY  -  A  collection  of  tests  given  to  students  before 
thev  begin  a  course.  The  battery  Is  designed  to  measure  student 
abilities,  attitudes.  Interests,  and  backgrounds,  (’reassessment 
data.  In  conjunction  with  W1 thl n-Course  Testing,  are  used  for 
Individualized  Instructional  assignment,  and  Student  Progress 
Management . 

PRI SCRIPT  ION  -  A  computer-generated  student  status  report  Indicating 

(a)  the  student's  performance  on  the  previous  assignment,  (h)  tfie 
next  assignment,  and  (c)  anv  training  resources  required  for  this 
assignment. 

RANDOM  ASSIGNMINT  -  lhe  option  to  specify  a  percentage  of  students  who 

should  be  randomly  assigned  to  the  alternative  modules  for  a  lesson. 
Provides  control  groups  for  AlS-related  research  and  lor  development 
of  regression  equations. 

Kl.fiRl  SS10N  liJOATION  -  Statistical  methodology  employed  In  calculating 
regression  models.  Ihe  mathematical  equation  which  provides  the 
best  prediction  (least  squares  fit)  of  expected  student  perfor¬ 
mance.  Regression  equations  are  based  on  the  performance  of 
prior  groups  of  students. 

R!  SObRCI  Al  I.OCAT ION  -  '  function  of  the  Adaptive  Model  for  managing  all 
training  resources  declared  In  the  Course  Mata  base  as  computer- 
managed.  i  fie  A1S  capability  to  balance  student  flow  through  a 
lesson  group,  block,  or  course  to  avoid  queueing  as  a  result  of 
resource  unaval labl 1 1  tv  and  to  maximize  use  of  critical  resources. 

SI  l  I  -PACI  P  CMI  CtMlRM  -  A  self-paced  course  suooorted  hv,  as  a  minimum, 
baseline  CMI. 

.11  (-PACING  -  A  generic  description  of  Instructional  programs  In  which 
learning  and  progress  noun  at  each  student's  sel f -es tab  1 1  shed 
pace . 

St'MWA1’1  -  See  Compute1  Software. 

SPI Cl  At  11  TRAINING  s I ANMARP  (SIS)  -  Air  force  establlsheu  standards  that 
specify  (nowledge  and  skill  requirements  for  training  In  Air  force 
specialty  Codes  and  are  the  primary  control  documents  tor  lvpe  111 
^  f  orma  1 res  1  dent )  courses  . 

si  All -Ml -Hi! -AR1  -  Current  level,  state,  or  condition  o’  technology  in 
dlsilpllnes  related  to  computer-based  education  ana  training. 


GLOSSARY  OF  TERMS  (Continued) 

STATISTICAL  SIGNIFICANCE  -  The  probability  that  the  observed  difference 
between  two  or  more  groups  could  have  occurred  by  chance  alone. 

STRATEGIES  -  Specific  Instructional  techniques  applied  within  a  module, 
lesson,  block,  or  other  sequence  of  Instruction  which,  considering 
the  particular  learning  objectlve(s),  are  designed  to  meet  the 
Individual  needs  and  characteristics  of  various  types  of  students. 


